What this covers

This is a survey talk by Jeff Dean, Google's Chief Scientist, given at Rice's Ken Kennedy Institute. It traces roughly fifteen years of machine learning as a sequence of enabling steps. It starts with the simple-techniques-at-scale era, including the "stupid backoff" language model trained on 2 trillion tokens. From there it moves to word embedding representations, where directions in vector space encode relations like king to queen, and then to sequence-to-sequence mapping for translation. It continues through the 2012 AlexNet result from Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, the Transformer, and conversational models, and ends at the Gemini family. The hardware thread runs alongside: specialized accelerators suited to ML's narrower computations, and a software layer that maps jobs onto pods of chips.

The middle of the talk covers how capabilities are drawn out of large models. It discusses chain-of-thought prompting, which asks a model to show intermediate steps and helps most on math and reasoning benchmarks at sufficient scale. It also covers Gemini's design as a multimodal model from the start, turning text, images, video, and audio into token sequences, and its second-place Elo rating on the LMSYS chatbot arena. The talk also raises the tradeoff between narrow, purpose-trained models and large multimodal base models with light fine-tuning, and it stresses evaluation as central to training decisions.

The applications section is the broadest. It includes phone features such as computational photography, call screening, and live captions. In science, learned simulators can run 100,000 times faster than conventional HPC simulation, which allows searches over millions of candidate materials. In health, it covers diabetic retinopathy screening, where models match the grade of the annotators they were trained on, and Med-PaLM, which went from passing board exams to expert-level performance in six months. It also considers AI tutoring set against the two-standard-deviation benefit of individual instruction. Two normative points are more contested: that training data reflects the world as it is, so automating biased processes accelerates unfairness, and that LLM focus may be crowding out other promising research directions.

Sharpest takeaway

Machine learning has fundamentally transformed what computers can do across vision, speech, and language through scale, specialized hardware, and algorithmic innovations like transformers, enabling multimodal systems that can understand and generate across diverse data types with significant practical applications.

  • Scale in data, compute, and model size consistently delivers better results across computer vision, speech recognition, and language tasks over the past decade
  • Specialized hardware (TPUs) and algorithmic breakthroughs (transformers) have enabled 10-100x efficiency gains and new capabilities
  • Multimodal models that can handle text, images, video, and audio simultaneously are the future, enabling applications from education to medical imaging

The claims · ranked36 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.80

Simple techniques applied over very large amounts of data are very effective; for example, the 'stupid backoff' algorithm ignored theoretically correct smoothing methods like Kneser-Ney smoothing in favor of simpler approximations that worked reasonably well when applied to 2 trillion tokens of data.

causalhigh valueestablishednovelty 2/4durability 4/4· Justin Basilico

“we came up with a an new algorithm called stupid backoff that kind of ignored the right mathematical thing to do and did something much simpler. So that when you looked up a 5 g and there was no data there, you would just look up the 4 g that was its prefix and use that if it was there and if it wasn't there, you'd look up the 3 g and so on. And then actually worked reasonably well compared to the fancier Kneser-Ney smoothing, which is what you really want to do, but is actually kind of computationally hard. Um so I mean one lesson from this is simple techniques over large amounts of data are very effective. This has been a lesson throughout my my career that you can actually do very simple things and the data speaks when you do that.”

0.75

Further refinement of general-purpose language models on domain-specific data can produce highly capable domain-specific models; Med-PaLM exceeded the medical board exam passing threshold, and Med-PaLM 2 (6 months later) achieved expert-level performance on medical board exams.

causalhigh valueestablishednovelty 2/4durability 3/4· Justin Dozat (or primary speaker)

“some of my colleagues took some of our earlier work on the Palm model and then the Palm 2 model, which is a general-purpose model kind of like uh trained on general text, and decided to enrich it and further train it on medical data. So, medical kind of questions and medical articles. And what they found was the Med-PaLM model, the first one, actually exceeded the medical passing mark for the medical boards. And that when you then, 6 months later, they trained it on the Med-PaLM 2 model to do Med-PaLM 2, they actually got expert-level performance on the medical boards uh for this particular task.”

0.75

Machine learning is increasingly influencing multiple aspects of science through automated exploration of hypothesis spaces and creation of rapid learned simulators that can be 100,000x faster than traditional HPC simulations, enabling searches over millions of possible materials/chemicals to identify promising candidates for synthesis.

causalhigh valueestablishednovelty 2/4durability 3/4· Justin Basilico

“material science is a pretty interesting area where, you know, basically machine learning is starting to influence lots and lots of aspects of of science. Um both through kind of automated explorations of interesting parts of a scientific uh hypothesis space or through you know, creation of very rapid simulators that are learned rather than sort of traditional high high large large-scale kind of HPC style computation. You know, in some areas you've been able to learn a simulator that is sort of the functional equivalent of a hand-coded simulator, but is now 100,000 times faster. And so, that means that all of a sudden you can, you know, search a space of 10 million possible chemicals or materials and identify ones that are interesting and promising and have certain properties that you would normally have to apply a lot more compute for.”

0.71

Multimodal models sometimes improve performance on all domains by providing additional context and understanding, though models trained on highly specific targeted datasets often perform well on narrow problems; for complicated or data-constrained problems, large multimodal base models with small amounts of fine-tuning data are preferable.

causalhigh valueestablishednovelty 2/4durability 3/4· Justin Basilico

“if you have a narrow problem and you collect a if you have a narrow problem and you collect a very targeted data set that is designed to tackle that just that problem, that will often, you know, give you good performance on a problem... if you have a complicated problem or it's hard to collect very specialized data, what you want is a model that has a huge amount of knowledge of lots of different things in the world, you know, from language and from, you know, images and audio, and then to be able to apply that model to the problem you care about. And then if you have a little bit of data for a problem you care about, then you're going to want to start with that base model and then fine-tune it or do in-context learning”

0.70

Sequence-to-sequence models use an encoder-decoder architecture where an encoder processes input sequences (e.g., English text) into a learned representation, which initializes a decoder that generates output sequences (e.g., French translation) one word at a time, and scaling this approach yields major improvements in translation accuracy.

factualhigh valueestablishednovelty 1/4durability 4/4· Justin Basilico

“you would have an input sequence. Uh let's take the case of translation. So, you put in an English sentence one word at a time and the system kind of builds a representation from its own current state plus the new word that it's now seeing and updates that state... you now have a distributed representation for the sentence you've seen so far... when you hit an end of sentence marker, you now train the model to spit out the correct the the translation of that sentence... use a neural encoder over this input sequence to initialize the state, which kind of gets you into the I've now absorbed the input sentence and I now want to decode a word at a time the correct translated sentence... use that to initialize the state of the neural decoder. You scale this up and it works. You get major improvements in translation accuracy.”

0.70

Evaluation is critical for model development: identifying strengths and weaknesses, monitoring training progress, making data composition decisions, and comparing capabilities to other models; this requires evaluating across diverse benchmarks and making online decisions about trade-offs.

normativehigh valueestablishednovelty 1/4durability 4/4· Justin Basilico

“evaluation, you know, really helps us identify the model strengths and weaknesses, helps understand, you know, is training going well? So, we're constantly evaluating these metrics as we're training the model. Um, helps us make decisions about what to change... there's a lot of complicated trade-offs. Some of which you make at the beginning of training, some of which you're kind of monitoring online and trying to make, [38:54] uh, principled or seat-of-the-pants decisions, I guess.”

0.69

Machine learning has changed expectations of what computers can do, moving from speech recognition that made many errors 10-15 years ago to systems that can seamlessly see, perceive, and understand the world around us in ways comparable to when animals evolved eyes in computing.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Basilico

“in recent years, I think machine learning has really changed our expectations of what we think of computers as being able to do. If you think back 10 or 15 years ago, you know, speech recognition kind of worked, but it wasn't, you know, really seamless... we've moved from that state to one where you actually expect computers to be able to sort of see and perceive the world around us in a much better way... think about when animals evolved eyes, you know, we're sort of at that stage in computing. Uh we now have computers that can see and sense and that's a completely different uh ballgame.”

0.69

Machine learning has significant potential in healthcare across medical imaging and diagnostics, ranging from 2D images to 3D volumes (MRI, CT scans) and pathology images, with work in diabetic retinopathy showing that models trained on board-certified ophthalmologist annotations can match ophthalmologist performance, and models trained on retinal specialist annotations can match retinal specialist performance.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Basilico

“I think there's huge potential for using machine learning in all aspects of health care. Really, we've been doing a fair amount of work in the space of medical imaging and diagnostics for quite a while... if you train a model based on you know trained ophthalmologists annotating images... you can actually train a model that is as effective as board certified ophthalmologists. If you then go on to get that same training data annotated by retinal specialists who have a lot more expertise and and experience in the these cases you can actually train a model that is on par with retinal specialists is kind of the gold standard of of care in this space”

0.69

Modern smartphones include invisible machine learning enhancements to camera features through computational photography including portrait mode, night sight, magic eraser, and various audio processing features, making ML benefits transparent to users while significantly improving device capabilities.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Basilico

“there's a lot of machine learning uh kind of invisibly helping people in various ways uh and particularly on phones. So, you know, a lot of camera features in modern smartphones have gotten significantly better over the years through combinations of computational photography methods and machine learning methods uh together. You know, so portrait mode where you make the background all blurry so you look all fancy in the foreground... night sight where you try to take an image in very low light conditions... magic eraser. So, if you actually understand images and you point at like one of the telephone poles and says make and say make these go away, then the system can do that.”

0.69

Google's training infrastructure uses a high-level software abstraction layer (pathways) that automatically maps computations onto available hardware pods, handling chip placement, topology, and inter-pod communication, freeing developers from manual distribution logic while maintaining performance visibility.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Dozat (or primary speaker)

“one of the things about our training infrastructure is we we wanted to be able to have a very scalable fabric that can deal with, you know, you specify very high-level uh, description of the computation you want and then have a a a system that then maps that computation onto the available hardware you have...For example, you might describe your computation as I have these two parts that I care about. I don't care where you put them and not let the underlying pathways software system that we've built decide where to put them...it knows where the chips are located and what the topology and bandwidth is between them.”

0.69

Large language models exhibit world knowledge (e.g., knowing TPUs are Google-developed specialized hardware) and can proactively offer information beyond what was asked (e.g., volunteering Python code and explanations when asked to reverse strings).

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Dozat (or primary speaker)

“the model has a fair amount of world knowledge. It knows that TPUs are, you know, basically I I've told you a lot of this. They're specialized hardware processors developed by Google to accelerate machine learning...the fact that someone asked a question and it did what it was asked, but then it also said, 'By the way, here's this thing called programming and some Python code, uh, and here's how you would do this'”

0.68

Chain-of-thought prompting—asking models to show their work step-by-step rather than jumping directly to answers—improves model accuracy and interpretability by encouraging the model to think through intermediate steps, with dramatic improvements in math and reasoning benchmarks where naive prompting produces poor results but chain-of-thought prompting dramatically improves accuracy at sufficient model scale.

causalhigh valueestablishednovelty 2/4durability 3/4· Justin Basilico

“asking models to show their work improves the accuracy of the model and also the interpretability. Uh and so, some of my colleagues came up with a technique called chain-of-thought prompting... if you ask a model, you know, give it you usually you give it an example of a kind of question you want it to answer and then the actual answer for that question, and then you ask it a new question... you ask it to answer that question... if you instead ask the model and show demonstrate to it, you know, how do you show your work?... the model actually, [34:44] mhm you know, elicits these sort of more incremental steps to get to the answer, and it gets it right... there's a pretty dramatic effect, right?... the quality of responses is pretty bad when you just give it sta- standard prompting, but at some point the model scale becomes large enough that all of a sudden when you ask it with chain of thought prompting, your accuracy shoots up quite a lot.”

0.68

Individualized tutoring shows outcomes two standard deviations higher than classroom instruction; AI tutoring systems could potentially approach this performance level, offering significant educational opportunities if deployed effectively.

factualhigh valueestablishednovelty 0/4durability 4/4· Justin Basilico

“we know that individualized tutoring has outcomes that are two standard deviations higher when you have a one-on-one human tutor for education than when you have a much uh broader scale classroom setting. So could we get close to that in terms of having individualized uh tutoring? I think I think that that possibility is within our collective grasp.”

0.68

Using ML at scale brings tremendous opportunities but also responsibility to ensure deployment is socially beneficial and does good in the world, requiring careful consideration of implications.

normativehigh valueestablishednovelty 0/4durability 4/4· Justin Basilico

“there's tremendous opportunity for sure. Uh but there's also a lot of responsibility. How do we sort of take this work forward, make sure that it's socially beneficial, uh, and really, uh, kind of do good things in the world with it?”

0.68

In 2012, the AlexNet paper by Krizhevsky, Sutskever, and Hinton achieved a 13% improvement in ImageNet accuracy (from ~51% to ~64%), which was remarkable and a major breakthrough because they were the only entrants using neural networks out of ~28 competitors, and this led nearly all subsequent entrants to adopt neural networks.

factualhigh valueestablishednovelty 0/4durability 4/4· Justin Dozat (or primary speaker)

“in 2011, the first year the contest was run, uh the winning entry got uh 50.9% accuracy. And then uh the next time the contest was run uh in a very famous landmark paper uh affectionately known as the AlexNet um by Alex uh Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, they made a giant leap forward uh in accuracy, you know, 13% or so improvement in accuracy, which was just remarkable. And they were the only one of about 28 entrants that year that used a neural network.”

0.66

Avoiding unfair bias in ML requires not just algorithmic debiasing techniques but awareness that training data reflects the world as it is, not as we want it to be; automating biased processes accelerates unfair patterns, so careful data curation and model evaluation for bias are essential.

normativehigh valuecontestednovelty 1/4durability 4/4· Justin Basilico

“avoid creating or reinforcing unfair bias. Often when you train these models, they're trained on uh wor- data from the real world and that's often the world as uh not the world as we'd like it to be, but the world as it is. And so it's really important when you're deploying machine learning models that you don't sort of train on data that it biased in unfair ways and then accelerate that because now you can automate and make these decisions more rapidly. Um so, there's a bunch of techniques you can apply uh on a sort of algorithmic basis to remove some kinds of bias. Um and what we strive to do is sort of apply the best known current techniques, but then also do research on advancing the state of the art in these areas”

0.66

Word embeddings—representing words as high-dimensional vectors learned through context prediction over trillions of tokens—capture semantic structure where similar words cluster together and directions in embedding space encode semantic relationships like the direction from king to queen being similar to the direction from man to woman, and going from present to past tense.

factualhigh valueestablishednovelty 1/4durability 4/4· Justin Basilico

“we're going to try to move words that appear in similar context nearer each other and we're going to try to push apart words that appear in in different context... you end up with really nice properties where in this 100-dimensional space... things that are very similar end up near each other. So if you have mountain and hill and cliff, they will all tend to be kind of near each other in that high-dimensional space... directions also are meaningful in this high-dimensional space... if you look at where king is in this space and you want to get to queen, you go in a certain direction... it turns out that king minus queen, that's the direction, is roughly the same as the direction you would go to get from from uh man to woman.”

0.65

Training large language models on conversational data using transformer architectures produces coherent dialogue, with evaluation methods ensuring sensible and specific responses rather than vague outputs, forming the foundation of modern chatbots like neural conversational models, Meena, ChatGPT, and Bard.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Basilico

“a group of people uh decided to train uh scale up and train on conversational style data using a transformer model instead of a recurrent model and that gave you know quite good results and in particular a way of evaluating those so that it's uh both sensible in what it responds and also specific. You don't want your chatbot to be overly vague like yeah, it's nice. You want it to actually say something sensible in response to to what you you interacted with it because that makes it more engaging and useful.”

0.65

Machine learning computations differ fundamentally from traditional handwritten C++ code in what they require from hardware; specialized hardware optimized for machine learning can focus on a narrower set of computations and do them extremely well and efficiently, enabling even larger scale with lower economic and energy costs.

causalhigh valueestablishednovelty 1/4durability 3/4· Justin Basilico

“the kinds of computations we want to run because of this new machine learning learning-based paradigms is pretty different than traditional handwritten twisty C++ code that, you know, a lot of basic CPUs were designed to write to to run effectively. And so we want different kinds of hardware in order to run these computations more efficiently. And we can actually, in some sense, focus on a narrower set of things we want computers to do and do them extremely well and extremely efficiently”

0.65

Gemini was designed as multimodal from the beginning, ingesting text, images, video, and audio and converting them into sequences of tokens that are processed by a transformer-based model, with separate decoding paths for text generation and image pixel generation that can be initialized from the transformer's learned state.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Basilico

“Gemini was really multimodal from the beginning. So, one of the things we did, as I mentioned, we didn't want it to just deal with text, we wanted to deal with with images and video and audio, and we turn that into a sequence of tokens that we then train a transformer-based model on. Uh and then we have a couple of different decoding paths. One we train to generate tokens uh and that are textual, and then the other we initialize the decoder with the state uh that the transformer has learned, and then we can sort of go from that state to a full uh you know, set of pixels for an image.”

0.64

Diabetic retinopathy is a degenerative eye disease that is highly treatable if caught early but can cause full or partial vision loss if untreated, and should be screened annually in anyone with diabetes or pre-diabetes, but many parts of the world lack sufficient ophthalmologists trained in retinal image interpretation.

factualhigh valueestablishednovelty 0/4durability 4/4· Justin Sazin

“diabetic retinopathy is a is a degenerative eye disease that can you know if you catch it in time is very treatable but if you don't you can suffer full or partial vision loss... people who are at risk which is sort of anyone with diabetes or pre-diabetes should be screened every year but in a lot of parts of the world there just aren't enough ophthalmologists to do the screening”

0.64

Gemini Pro achieved the second-highest Elo rating on the LMSYS chatbot arena (which evaluates models via user-submitted prompts and anonymous pairwise comparisons across ~30 models), demonstrating competitive performance against other state-of-the-art chatbots.

factualhigh valueestablishednovelty 1/4durability 2/4· Justin Dozat (or primary speaker)

“we actually put Gemini Pro in Bard uh in {slash} Gemini last month and there's a public site called LMSYS that can evaluate different chat agents cuz there's now a lot of different chatbots in the world...the pro level model actually achieves the second highest Elo score in this thing and there's probably like 30 different models.”

0.63

Phone features for audio/text transformation include screen call (computer-generated voice answering calls and providing transcripts), hold for me (listening to hold music while waiting), and live caption (transcribing video audio in real-time), providing accessibility and convenience benefits.

factualhigh valueestablishednovelty 0/4durability 3/4· Justin Basilico

“sometimes uh you want to be able to say screen a call. Uh so, maybe you don't want to actually answer your phone but have a uh computer-generated voice answer the phone for you, ask what the person's calling about, and then give you a transcript of what they said... hold for me can kind of listen on the phone for you so you don't have to hold uh hold on Bank of America when you're calling their customer support. Live Caption can show can take any video playing on your phone and listen to the audio and then give you transcripts uh captions of what's being narrated.”

0.62

LLM-focused research may be crowding out other innovative ML approaches; showing even at small scale that alternative approaches work well can draw attention to under-explored directions.

factualhigh valuecontestednovelty 1/4durability 3/4· Justin Basilico

“are we crowding out other innovative ideas that maybe uh are you know, not as fully developed and so they don't look as good as some of the things that have been, you know, much more fully explored”

0.61

The first TPU generation (TPU V1) achieved a 30 to 80x improvement over CPUs in both energy efficiency and computational performance for inference tasks.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Basilico

“the first one we built was for inference... we built uh the first generation, which was really just a single-card system uh that had one of these accelerators on it, TPU V1. Um And that uh actually was a, you know, about a 30 to 80x improvement over using a CPU at the time in terms of both uh energy efficiency and and computational performance.”

0.61

TPU pod networks use very simple 2D mesh topologies where each chip connects to its four neighbors with direct wires, enabling very high-speed bandwidth and low-cost connections without routing overhead, with the first generation connecting 256 chips in a 16x16 grid.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Basilico

“they have very very simple, but high bandwidth networks uh, in the pod. So, basically each chip in this uh, first generation was connected to its four neighbors in a 2D mesh. So, you have a 16 by 16 grid of chips in some sense in these in these racks. And every chip is connected to its neighbor with basically a wire. Uh, and so, that means you don't have to do any routing in the network. Um, and so, you can have very high-speed bandwidth, very low-cost connections because you're only trying to go 6 in to the next chip or something like that.”

0.61

Multimodal reasoning in Gemini enables solving complex multi-step physics problems: given a handwritten student solution and problem image, the model can identify errors (e.g., using hypotenuse instead of height in potential energy calculation), explain the mistake, and provide the correct solution with LaTeX formatting.

factualhigh valueestablishednovelty 1/4durability 3/4· Justin Dozat (or primary speaker)

“here's a solution to a physics problem by a student, and then there's just a uh a picture of the problem and the student's written out answer in kind of handwriting...this is the output of the model. So the student get didn't get the correct answer. Student made a mistake in the calculation of potential energy at the start of the slope. Um the potential energy at the start is given by MGH. Student used the length of the slope, I guess that's the hypotenuse, instead of the height in the calculation. Correct solution is...Therefore we can write if this is this is actually in LaTeX...substituting the values, there it is. It worked the problem out to two decimal places.”

0.61

Data quality is a critical research area; high-quality data makes a huge difference in model performance on tasks, potentially more important than model architecture itself; important research directions include automatically learning curriculums and identifying high-quality versus low-quality examples.

normativehigh valuecontestednovelty 2/4durability 3/4· Justin Basilico

“I do think data quality is a really interesting and important research area. Um and I think it's we've seen that, you know, having really high-quality data makes a huge difference in the performance of the model on tasks you care about. And so that means, you know, in some sense that's as important or even more important in some cases than the actual say model architecture you're using. Um and and so, you know, I think it's a pretty important area for future research. You know, having the ability to learn curriculums automatically seems important. Uh identifying high-quality examples and low-quality examples and so on.”

0.59

Large language model parameter counts have grown exponentially: GPT-2 had 1.5 billion parameters (2019), T5 had 11 billion, GPT-3 had significantly more, and Gemini (co-led by the speaker) continues this progression with multiple sizes including Ultra, Pro, and Nano variants optimized for different deployment scenarios.

factualhigh valueestablishednovelty 0/4durability 3/4· Justin Basilico

“there's a progression of neural language models... GPT-2 from OpenAI which was you know, some of these have parameter counts which you can think of as a rough sense of the scale of the model. So, 1 and 1/2 billion parameters in 2019, the T5 work from Google from some colleagues of mine uh 11 billion parameters... Gemini, which is the the project I co-lead with my colleague Oriol Vinyals... we have a large collection of people in lots of different uh research offices working on building capable uh multimodal models... the V1 generation of Gemini comes in three different sizes. So, Ultra is the kind of the largest scale uh, and most capable model we have. Pro is like a good size for running in a data center uh, and we use that in a lot of different product contexts... the Nano model, you actually want a lot of these machine learning models to be able to run on device.”

0.56

Large-scale model training failures are inevitable; minimizing failures and recovery time is critical, with optimization including coordinated kernel upgrades across entire machines clusters simultaneously (rather than rolling updates) and rapid recovery from in-memory checkpoints instead of distributed file systems, achieving recovery in 5-10 seconds rather than minutes.

causalhigh valuespeaker onlynovelty 2/4durability 4/4· Justin Basilico

“as you scale up, you know, failures will happen... minimizing failures is really important... we had a a sweeping way of upgrading kernels on our machines, which is a perfectly fine approach if those machines are kind of independent computations, but if they're all part of the same, you know, thousand machine computation, you actually would prefer to take the machines down, upgrade all thousand kernels simultaneously and then bring them back up rather than having rolling failures throughout... you then want to minimize the time to recover uh because the faster you you can recover, the the sooner you can actually be making uh useful forward progress... we have a a metric we call goodput, which is the percentage of time that model training is actually making useful forward progress... rapid recovery from other copies of the model state from memory in those other machines rather than going to a distributed file system to recover from a checkpoint. And that, you know, makes the recovery time, you know, a matter of a few 5 to 10 seconds rather than several minutes.”

0.55

Gemini Ultra exceeded state-of-the-art performance on 30 of 32 academic benchmarks evaluated, including achieving 90% accuracy on MMLU (a 57-domain expert-level benchmark) which exceeds human expert-level performance at 89.6-89.8%, and achieving state-of-the-art on diverse benchmarks in text reasoning, math, coding, image understanding, video understanding, audio, and multilingual tasks.

factualhigh valueestablishednovelty 0/4durability 3/4· Justin Basilico

“the highest-level summary is, you know, we looked at 32 academic benchmarks. Uh, and the Gemini Ultra model exceeded the state-of-the-art performance in 30 of the 32... the ones in blue is the state-of-the-art. And so, we we state-of-the-art on seven of the eight. Uh, the 90% on MMLU is interesting because this is a very broad uh set of questions in 57 different subjects... the group that put together the the benchmark, um, measured human expert-level performance at 89.6%, I think, or maybe 89.8. And so, this actually exceeds human expert-level performance in these 57 categories”

0.51

Large amounts of untapped training data remain: video data has been trained on only in small amounts relative to available video, and understanding the world through visual and audio data will be different from and complementary to training on language, requiring both modalities.

forecasthigh valuecontestednovelty 1/4durability 2/4· Justin Basilico

“I think we've not really begun to train on, say, video that much. I mean, we've done small amounts of video, but there's a huge amount of video data in the world. I think actually understanding the world through visual and audio data will be different than sort of training on a lot of language. You're going to want to do both. But, I don't think we've we've really exhausted the training data in the world.”

0.47

The speaker prefers not to use the term 'LLM' because the field is moving toward multimodality; true multimodal systems will include not just human modalities (vision, audio, language) but also up to 50-100 other modalities important in the world like time series data and sensor data for healthcare applications.

normativehigh valuespeaker onlynovelty 2/4durability 3/4· Justin Basilico

“I would say mo- you know, I I tend not to use LLM because I think we're moving to a multimodal world. Uh and I think multimodal is going to be more than kind of the human modalities you think about like uh visual and audio and language but other modalities that are important in the world you know, like time series of interesting, you know, heart rate sensor data for health care applications. There's probably 50 to 100 modalities of data you'd want to be able to deal with.”

0.45

Generative image models can produce diverse images from detailed text prompts, with scaling improving details and text rendering fidelity; images are generated in progressive stages—first a small low-resolution image is created from text embedding, then progressively upsampled while conditioned on both the lower-resolution image and text embedding.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Justin Basilico

“you put in a prompt that gives you a representation of what that sentence in a distributed vector-based setting is, and then conditioned on that, the model is trained to generate first a small-scale image, and then take uh, another a model that is designed to improve the res- increase the resolution of an image, uh, conditioned on both that lower-scale, lower-resolution image plus the text embedding, and then we apply that one more time with the larger image and the conditioned on the text embedding to produce the full-scale 1024 by 1024 image.”

0.43

Google published ML principles in 2018 to guide internal teams on applying ML responsibly, including avoiding unfair bias by not automating and accelerating biased patterns from training data, improving interpretability and accountability, respecting privacy, and ensuring social benefit.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Justin Basilico

“as we've gone from doing basic research in in machine learning to then using it in a lot of places in all of our products, we started to think about a, you know, a set of principles by which we want to, you know, think about the implications of using machine learning, what considerations should we have for, you know, various ways in which we might apply it. Um and we in 2018 published a set of principles that we came up with. Um really these were designed to help ed our own uh internal teams about machine learning and things you should be thinking about”

0.41

When asked to estimate the number of companies per million residents in various countries (US, England, South Korea, Taiwan, Singapore) and present results in table form, Gemini correctly provides a table with estimates, interprets the results (ranking by number of companies), acknowledges uncertainty ('these are just estimates'), and when asked for sources, provides specific databases used for each country with appropriate caveats about different definitions.

factualhigh valuespeaker onlynovelty 1/4durability 1/4· Justin Sazin

“he was asking about please estimate the number of companies per million residents... And so Gemini said absolutely here's a table... England has the highest number followed by the US... It's important to note these are just estimates. The actual number may vary... where did you find the numbers... So the company numbers I used came from a combination of different sources. So for the US they used the Dun & Bradstreet database... Please note these sources have somewhat different definitions.”