multimodal models
Models that process multiple input types (e.g. image and text), with potential to include learned task-specific modalities.
As defined in Fully autonomous robots are much closer than you think – Sergey Levine
Other definitions
AI models trained on and capable of processing multiple types of input data (text, images, video, audio) together, enabling cross-modal understanding
Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
Where it comes up · 6
- 326 - Ai Information Integrity2 claims
- OpenAI CEO Sam Altman | AI for the Next Era2 claims
- Marc Andreessen on AI, Geopolitics, and the Regulatory Landscape | Ray Summit 20241 claim
- Fully autonomous robots are much closer than you think – Sergey Levine1 claim
- Dwarkesh Patel Podcast - The most important question nobody's asking about AI1 claim
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold1 claim
Most important claims · 8 of 8Search all claims for “multimodal models” →
- Generative AI enables hyper-personalization, an 'audience of one,' where everyone has a private conversation with an oracle that caters to their preferences and biases, producing a balkanization of worldview in which no one sees the same information (e.g., a Wikipedia article curated solely for you), accelerating the loss of mutual interpretability already visible on social media.326 - Ai Information Integrity
- Research (cited as from the Middlesbrough institute of terrorism testing GPT-3) tested how an early form of GPT performed as a radicalization agent, and as chatbots become more sophisticated and multimodal they could groom or radicalize individuals by entrenching their worldview through generated text, audio, and video.326 - Ai Information Integrity
- Large multimodal models can treat different modalities as tokens (words, pixels, actions, etc.), enabling transfer across domains; training such systems is harder than pure language models, but the resulting multimodality produces better performance across all modalities as they learn from each other's structure.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- Multimodal AI systems integrating text, images, and all other data modalities into single unified models will be developed and will unlock new applications, unlike current systems that treat modalities separately.OpenAI CEO Sam Altman | AI for the Next Era
- True multimodal models incorporating every modality in one system that can fluidly move between different modalities will be achieved and will unlock massive applicationsOpenAI CEO Sam Altman | AI for the Next Era
- Processing every one of the ~100 million US CCTV cameras with cheap open-source multimodal models (one 1,000-token frame every ten seconds) costs about $30 billion per year today, but since a given AI capability gets 10x cheaper year over year, by 2030 it could be cheaper for the government to understand every nook and cranny of the country than to remodel the White House.Dwarkesh Patel Podcast - The most important question nobody's asking about AI
- The key to overcoming context limits is representing context in the right form for the goal, varying by modality, since humans hold some context symbolically (a shopping checklist as words) and other context spatially or visually (navigating a route), and multimodality has far more potential than image-plus-text alone.Fully autonomous robots are much closer than you think – Sergey Levine
- There is a software wave happening to merge AI approaches by creating multimodal models capable of reasoning, planning, and potentially powering robots—OpenAI's 01 is a reasoning engine, Llama release has open source multimodal capabilities, and within a year a single multimodal model could power robot reasoning and planning.Marc Andreessen on AI, Geopolitics, and the Regulatory Landscape | Ray Summit 2024