Multimodal (models)
Machine learning models that can process and generate multiple types of data simultaneously (text, images, audio, video, etc.) by converting all modalities into a unified token sequence for a single architecture, enabling cross-modal reasoning and unified learned representations.
As defined in Jeff Dean (Google): Exciting Trends in Machine Learning
Where it comes up · 1
Most important claims · 5 of 5Search all claims for “Multimodal (models)” →
- Multimodal models sometimes improve performance on all domains by providing additional context and understanding, though models trained on highly specific targeted datasets often perform well on narrow problems; for complicated or data-constrained problems, large multimodal base models with small amounts of fine-tuning data are preferable.Jeff Dean (Google): Exciting Trends in Machine Learning
- Gemini was designed as multimodal from the beginning, ingesting text, images, video, and audio and converting them into sequences of tokens that are processed by a transformer-based model, with separate decoding paths for text generation and image pixel generation that can be initialized from the transformer's learned state.Jeff Dean (Google): Exciting Trends in Machine Learning
- Multimodal reasoning in Gemini enables solving complex multi-step physics problems: given a handwritten student solution and problem image, the model can identify errors (e.g., using hypotenuse instead of height in potential energy calculation), explain the mistake, and provide the correct solution with LaTeX formatting.Jeff Dean (Google): Exciting Trends in Machine Learning
- Large language model parameter counts have grown exponentially: GPT-2 had 1.5 billion parameters (2019), T5 had 11 billion, GPT-3 had significantly more, and Gemini (co-led by the speaker) continues this progression with multiple sizes including Ultra, Pro, and Nano variants optimized for different deployment scenarios.Jeff Dean (Google): Exciting Trends in Machine Learning
- The speaker prefers not to use the term 'LLM' because the field is moving toward multimodality; true multimodal systems will include not just human modalities (vision, audio, language) but also up to 50-100 other modalities important in the world like time series data and sensor data for healthcare applications.Jeff Dean (Google): Exciting Trends in Machine Learning