generative pre-trained transformer (GPT)
A type of large language model that is first pre-trained on vast text data to learn patterns and then fine-tuned for specific tasks, capable of generating new text
As defined in The Future of Artificial Intelligence
Other definitions
A class of large language models that are pre-trained using self-supervised next-token prediction on large text corpora, then fine-tuned for specific tasks, representing the dominant approach to building powerful language models
Yann LeCun's $1B Bet Against LLMs [Part 1]
Where it comes up · 2
Most important claims · 10 of 19Search all claims for “generative pre-trained transformer (GPT)” →
- One of Mitchell's biggest fears is that humans will trust AI systems with tasks they are not capable or robust enough to handle, as shown in Harvard Business School research where consultants using GPT-4 performed worse on quantitative reasoning tasks than those not using AI, because they over-relied on the system and made more mistakes.The Future of Artificial Intelligence
- AI systems trained on data from society full of biases will magnify those biases; facial recognition systems are much worse on people of color resulting in false arrests; chatbots trained on internet data produce racist information; and image generation systems cannot create images of Black African doctors treating white kids because it's too far from training data patterns.The Future of Artificial Intelligence
- GPT-1's success with a two-stage training process—extensive self-supervised pre-training on 7,000 books followed by supervised fine-tuning on labeled language tasks—proved that self-supervised pre-training could break models free from dependence on human-labeled data at scale.Yann LeCun's $1B Bet Against LLMs [Part 1]
- GPT-2 (2019) and GPT-3 (2020) were created by aggressively scaling up the GPT-1 approach, with GPT-3 achieving state-of-the-art results across a much broader range of language tasks through sheer scale.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Fine-tuning models through tools like LoRA (Low-Rank Adaptation) and access to open-source models allows individuals to customize large language models on consumer-grade hardware, creating a tension between democratic access and safety risks; companies and people worry this could enable malicious actors to fine-tune models for dangerous purposes.The Future of Artificial Intelligence
- We face 'radical uncertainty' about the future of AI because experts don't have good answers to fundamental questions like whether AI will revolutionize medicine, become smarter than humans in all cognitive tasks, replace jobs, destroy democracy, or cause human extinction.The Future of Artificial Intelligence
- Mitchell's research group tested abstract reasoning puzzles where humans are shown three demonstrations of a grid transformation and asked to identify which test grid follows the same pattern; humans achieved about 90% accuracy but GPT-4 only achieved about 33% accuracy on 480 such puzzles, suggesting GPT-4 lacks basic abstract reasoning abilities.The Future of Artificial Intelligence
- AI will fuel disinformation and scams through chatbots writing health care misinformation blogs, voice cloning to impersonate loved ones for scams, and AI-generated media tsunami of disinformation during elections.The Future of Artificial Intelligence
- The Transformer architecture, initially designed for language translation by Google, was repurposed by OpenAI for next-token prediction on self-supervised pre-training, becoming the foundation for GPT and all modern large language models.Yann LeCun's $1B Bet Against LLMs [Part 1]
- ChatGPT produces hallucinations—confident false statements—like claiming Mitchell wrote a book that didn't exist; these happen because the systems are trained to predict plausible text, not to tell the truth, and have no grounding in actual facts.The Future of Artificial Intelligence