Foundation model
A large model trained on massive general data that can be adapted to many downstream tasks; the term encompasses LLMs and similar large generative models.
As defined in 280 | François Chollet on Deep Learning and the Meaning of Intelligence
Other definitions
A large, pre-trained AI model (like GPT-4, Gemini, Claude) that is trained on diverse data and can be fine-tuned or adapted for many downstream tasks; training costs for foundation models are billions of dollars.
Eric Schmidt and Craig Mundie with David Rubenstein: GenesisA large language model trained on broad data that can be adapted for multiple downstream tasks, such as GPT-4, Claude, or Llama
Vertical AI Agents Could Be 10X Bigger Than SaaS
Where it comes up · 37
- Gaurav Misra & Dwight Churchill - Building Captions - [Invest Like the Best, EP.405]6 claims
- The future of intelligence | Demis Hassabis (Co-founder and CEO of DeepMind)6 claims
- Sir Demis Hassabis on The Future of Knowledge | Institute for Advanced Study6 claims
- The AI Reset Is Here: Search, Jobs, and Everything Else w/ Anish Acharya, Dave Blundin, Salim Ismail5 claims
- David Vélez - Building Nubank and the Future of Global Banking - [Invest Like the Best, EP.495]4 claims
- Breakthrough potential of AI | Sam Altman | MIT 20234 claims
- The Shape of AI to Come! Yann LeCun at AI Action Summit 20254 claims
- Mark Cuban3 claims
and 29 more
Most important claims · 10 of 78Search all claims for “Foundation model” →
- As models become more efficient and cheaper to train, including locally on a PC or phone, there will be millions of specialized AI models rather than four or five dominant foundation models — assembled via open source, copy-paste, or mixture-of-experts meta front ends — because brands like Mayo Clinic and Harvard Medical School will license or keep their own IP rather than donating it to ChatGPT or Gemini.Mark Cuban
- Every company training foundation models employs several thousand people full-time to create and rate training data by correcting answers to queries the model handles poorly; this manual annotation is what actually makes the models useful, beyond mere next-word prediction across internet text.280 | François Chollet on Deep Learning and the Meaning of Intelligence
- Foundation models are good at giving directionally accurate answers that point in the right direction but are very bad at giving exactly correct answers, especially for quantitative problems, so they should be used as a stepping stone toward a solution that you then refine yourself or with a symbolic search system, never blindly trusted.280 | François Chollet on Deep Learning and the Meaning of Intelligence
- Modern foundation models need only a small amount of task-specific data, around 50,000 examples, compared with millions required under older machine learning paradigms, so getting data is less of a barrier to building AI products.Chris Pedregal - Building Granola - [Invest Like the Best, EP.412]
- The fundamental trade-off of forcing AI representations to match a rigid isotropic Gaussian distribution is that the mandatory geometric complexity causes LayWM to fail on simple tasks with low intrinsic dimensionality, such as a 2D dot navigating through a door (two-room environment).LeWorldModel: Stable End-to-End JEPA World Models From Pixels
- The inevitable direction of history is that the vast majority of AI systems will be built on top of open source platforms, because only open source enables an ecosystem of companies and groups to fine-tune specialized systems that are diverse in language, culture, and value systems for vertical applications.Research Pioneer
- The dominant AI labs assume reasoning will emerge automatically from bigger models trained on more English text; Tenev says that may partly work but he sees it as an incomplete path.Vlad Tenev - Navigating Robinhood's Evolution - [Invest Like the Best, EP.384]
- Multimodal AI systems (processing text, code, video, and images simultaneously) are necessary for building world models and universal digital assistants that can understand context and assist with real-world tasks beyond text generation.Sir Demis Hassabis on The Future of Knowledge | Institute for Advanced Study
- There is a double standard in AI bias outrage: when Elon's Grok gave 'woke' answers he said it was a fixable mistake and was forgiven, but when Google's Gemini produced racially diverse historical images the response was that it was 'the end of the world' and a reflection of Google's culture, even doxxing the product manager — despite Gemini not even being a fully public product.Mark Cuban
- The AI sector will evolve into a single-digit number of very large-scale foundation models—'concentric circles' that cover massive overlapping capability with slightly differentiated edges where human value stacks on AI value—serviced by a much larger set of wrappers and specialized helper systems that change the form factor by which models integrate into life.Jack Clark