GPT-3
A large language model that takes a text prompt and predicts what words might follow based on statistical correlations in its training corpus, producing fluent but ungrounded text.
As defined in 184 | Gary Marcus on Artificial Intelligence and Common Sense
Where it comes up · 15
- 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT10 claims
- 184 | Gary Marcus on Artificial Intelligence and Common Sense6 claims
- “Deep Thoughts”: Tyler Cowen on AI and the Future of Work6 claims
- $11B AI CEO: Why You Must Maximize Stress to Win3 claims
- The Future of Knowledge Graphs in a World of Large Language Models3 claims
- Finding Undiscovered Talent2 claims
- Amjad Masad - The Future of Software Creation - [Invest Like the Best, EP.310]2 claims
- 151 | Jordan Ellenberg on the Mathematics of Political Boundaries2 claims
and 7 more
Most important claims · 10 of 41Search all claims for “GPT-3” →
- So far, LLMs had been trained using web data which gave them solid understanding of language semantics, syntax, and generation of fluent sentences, but GPT-3's objective was to predict likely sentences with no notion of correctness or factual bias, whereas the human goal in using an LLM is to generate coherent, informative, trustworthy, and reliable outputs.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Research on how knowledge is stored in large language model parameters is beginning to understand why models must be so large; Stable Diffusion 1 has only 890 million parameters (200x smaller than GPT-3's 175 billion) because it doesn't need to internalize knowledge of individual entities.The Future of Knowledge Graphs in a World of Large Language Models
- A GPT-3 model fine-tuned on roughly 2 million words of Daniel Dennett's writing ('DigiDan') produced answers to philosophical questions that Dennett experts could only distinguish from Dennett's real answers about 50% of the time (versus 20% chance), and on one question Dennett himself conceded the model's answer was arguably better.262 | Eric Schwitzgebel on the Weirdness of the World
- GPT-3 showed that large models can be few-shot or zero-shot learners, where few-shot learning refers to the ability where the model is shown some demonstrations or examples of a task during inference as conditioning.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- GPT-3 demonstrated one-shot and zero-shot learning abilities, and unlike fine-tuning, few-shot learning does not update the network's weights but instead just conditions the model during inference with the prompt.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Solving continual learning won't be a singular one-and-done achievement but will resemble in-context learning: just as GPT-3 demonstrated powerful but unfinished ICL that required years of further progress, labs will release something they call continual learning next year that counts as real progress, but human-level continual learning may take another 5 to 10 years.An audio version of my blog post, Thoughts on AI progress (Dec 2025)
- GPT-3's toxic-speech and truthiness problems cannot be solved with band-aids because it only has correlations, not a queryable underlying database; you cannot ask it whether it is making a toxic remark or fabricating anti-vaccine content because it has no representational scheme of entities and properties to query.184 | Gary Marcus on Artificial Intelligence and Common Sense
- GPT-3 may be a potential next platform because it is so powerful and amorphous, a generative model trained on a large chunk of the internet, including public GitHub, that can take text, code, or image inputs and may be capable of things no prior model could do.Lessons on Early Stage Investing and Getting Acquired
- OpenAI's GPT-3 language learning model can produce articles that look like they were written by journalists and are appearing with increasing frequency across the web, putting journalism jobs at risk.Will AI Take Your Job? AI Automation Explained
- GPT-3 represents a fundamental breakthrough in language models that will reorganize how institutions aggregate and deliver information within the next few years, as users shift from requesting links (Google) to requesting answers (AI), with cascading effects on internet marketing and knowledge distribution.“Deep Thoughts”: Tyler Cowen on AI and the Future of Work