multimodal model
A model trained on more than one type of data (e.g., both text and images) with a vector space jointly encoding the modalities.
As defined in 230 | Raphaël Millière on How Artificial Intelligence Thinks
Other definitions
A model that accepts and generates multiple data types such as text, images, audio, and video.
Howie Liu - Building Airtable - [Invest Like the Best, EP.375]A model that processes and generates multiple data types such as text, images, audio, and video.
Bret Taylor - The Agent Era - [Invest Like the Best, EP.386]
Where it comes up · 4
- Howie Liu - Building Airtable - [Invest Like the Best, EP.375]2 claims
- 230 | Raphaël Millière on How Artificial Intelligence Thinks2 claims
- Jeff Dean (Google): Exciting Trends in Machine Learning2 claims
- Geoffrey Hinton - Two Paths to Intelligence1 claim
Most important claims · 7 of 7Search all claims for “multimodal model” →
- Projects like Google's SayCan plug pretrained language models into robots to translate natural-language instructions into actionable sub-tasks using a vision-language model, but the key limitation is that these systems are not trained end-to-end through world interaction—they bolt together passively-pretrained modules—raising a deep question about whether such modular stitching, versus learning from the ground up via rich multimodal interaction, is the right model of cognition.230 | Raphaël Millière on How Artificial Intelligence Thinks
- Single neurons in multimodal models can be concept-specific and cross-modal at different abstraction levels—for example, a 'spider neuron' in the original DALL·E was preferentially activated by images of spiders, the word 'spider,' and images of Spider-Man, analogous to the famous 'Jennifer Aniston neuron' findings in human neuroscience.230 | Raphaël Millière on How Artificial Intelligence Thinks
- Multimodal models sometimes improve performance on all domains by providing additional context and understanding, though models trained on highly specific targeted datasets often perform well on narrow problems; for complicated or data-constrained problems, large multimodal base models with small amounts of fine-tuning data are preferable.Jeff Dean (Google): Exciting Trends in Machine Learning
- In this sense, I think these AI models can have subjective experience. Suppose we make a multimodal model with a camera. Without its knowledge we put a prism in front of the camera. Then we put an object in front of it and ask where the object is. It says the object is there (pointing). We say you're wrong, there's a prism. The system says 'I have subjective experience of the object being there.' This is the same use of subjective experience we use for people.Geoffrey Hinton - Two Paths to Intelligence
- Current LLMs already have the pieces of a much more advanced intelligence; the main gap is learning to chain them together, and reasoning quality, not novel capabilities, will be the most important advancement.Howie Liu - Building Airtable - [Invest Like the Best, EP.375]
- Large language model parameter counts have grown exponentially: GPT-2 had 1.5 billion parameters (2019), T5 had 11 billion, GPT-3 had significantly more, and Gemini (co-led by the speaker) continues this progression with multiple sizes including Ultra, Pro, and Nano variants optimized for different deployment scenarios.Jeff Dean (Google): Exciting Trends in Machine Learning
- A single model that accepts full video, long reports, documents, text, and audio and outputs any of these is coming very soon, but it is less profound than reasoning gains because value can be piecemealed today through specialized models integrated with data and workflows.Howie Liu - Building Airtable - [Invest Like the Best, EP.375]