
Yann LeCun's $1B Bet Against LLMs [Part 1]
What this covers
This is the first part of a two-part Welch Labs explainer on Yann LeCun's case for moving past large language models. It begins with LeCun's early convolutional nets, which resembled AlexNet 25 years before AlexNet, and his "cake" analogy for intelligence: self-supervised learning is the bulk, supervised learning the icing, and reinforcement learning the cherry. From there it traces how self-supervised pre-training took over language. The sequence runs from the Transformer, repurposed by OpenAI for next-token prediction, through GPT-1 trained on 7,000 books to GPT-2 and GPT-3 by scaling. Reinforcement learning's mid-2010s renaissance is treated as a detour with limited reach.
The middle of the video asks why the same recipe has not worked for vision. Generative video models give blurry output because they must average over an astronomically large set of possible next frames, which a language model avoids by enumerating discrete tokens. The alternative is the joint embedding architecture, which goes back to the Siamese networks LeCun's team built at Bell Labs for detecting forged signatures. Its central obstacle is representation collapse, where a network outputs the same embedding for everything. The video covers contrastive fixes and their computational cost, then the Barlow Twins method, which draws on Horace Barlow's 1961 redundancy-reduction hypothesis. VICReg and DINO follow, including DINO's label-free segmentation and DINO V3's 88.4% ImageNet accuracy, set against a Vision Transformer's 88.6% in 2021.
The final stretch turns to JEPA and world models. JEPA predicts future embeddings instead of pixels, and inference becomes a search over predicted futures. V-JEPA 2 conditions on robot-arm actions so it can plan toward a goal state. The contested claims sit here. LeCun holds that LLMs are limited to domains where language is the substrate of reasoning and cannot model the physical consequences of actions. He also argues that a teenager learns to drive in about 20 hours while AI systems remain stuck at Tesla's Level 2 autonomy. The video closes by asking whether JEPA actually performs well.
Yann LeCun argues that joint embedding predictive architectures (JEPA) represent a superior alternative to large language models for building intelligent systems because they avoid generative modeling's fundamental limitations and can learn world models that predict consequences of actions, which is essential for true AI agency.
- Generative video prediction models produce blurry outputs because they must predict one of 10^15 million possible frames, making pixel-level reconstruction intractable
- Joint embedding architectures with Barlow Twins and related methods solve representation collapse and have achieved near state-of-the-art image classification without labels
- World models built on JEPA enable planning through action prediction, a capability LLMs fundamentally lack
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Large language models are fundamentally limited to language manipulation and reasoning only in domains where language itself is the substrate of reasoning, making them incapable of building world models that predict physical consequences of actions.
“LLMs are really good at manipulating language, but basically nothing else. Um and they're really good in domains where the language itself is the substrate of reasoning.”
Generative video prediction models that directly predict pixel values produce blurry outputs because they cannot enumerate the ~10^15 million possible next frames the way language models enumerate discrete tokens, forcing them to average pixel values across uncertain outcomes.
“For full HD video in the most general case, each pixel can take on 256 discrete values, and we have 1920 * 1080 * 3 color pixels, meaning there are something like 10 to the power of 15 million possible next video frames, dwarfing the number of atoms in the observable universe.”
Inference in JEPA-based systems becomes a search process over predicted futures rather than autoregressive prediction, fundamentally changing how these systems generate behavior.
“And the inference process now becomes a search as opposed to just a auto-regressive prediction.”
Representation collapse in joint embedding architectures—where networks trivially output identical embeddings for all inputs—was the primary unsolved problem preventing joint embeddings from learning useful representations until Barlow Twins and related approaches.
“Since we're training our network to make the embeddings of our original and corrupted images or videos as similar as possible, the network can find a trivial solution where it simply returns the same embedding vector for any input that we pass in. If our network learns to output, for example, a vector of all ones for any input, then the network will return all ones for a corrupted and non-corrupted view of the same image, maximizing the resulting similarity, but without actually learning anything useful. This problem is known as representation collapse.”
JEPA (Joint Embedding Predictive Architecture) avoids the pixel prediction problem by mapping observations to embeddings and training a predictor to forecast future embeddings rather than pixels, allowing the model to focus on salient scene features instead of random unpredictable details.
“JEPA means joint embedding predictive architecture, and it's you take an observation in the world, and then the next observation in the world, uh you run them through encoders, so it's like a joint embedding type architecture, and then you have a predictor that tries to predict that the state at time t + 1 from the state at time t”
VJEPA-2 conditions JEPA on action signals sent to a robot arm, allowing the predictor to learn how control signals change the robot's position in embedded space, enabling planning by exploring the model's predictions to find actions matching a goal state.
“In the VJEPA 2 paper, the team conditions a JEPA model on the action signals sent to a robot arm. So, the JEPA model sees a sequence of images of the robot's arm and environment, and then is trained to predict the embedding of the next video frame, but is also given the control signals that are sent to the robot arm. This allows the predictor to learn to predict how various control signals will change the robot arm's position in the embedded image.”
GPT-1's success with a two-stage training process—extensive self-supervised pre-training on 7,000 books followed by supervised fine-tuning on labeled language tasks—proved that self-supervised pre-training could break models free from dependence on human-labeled data at scale.
“Radford and his OpenAI colleagues trained their transformer on a fairly large internal OpenAI data set of 7,000 books. Note that we now call this phase pre-training. And then further trained their model using standard supervised learning from human-generated labels on specific language tasks. Their two-stage training approach worked well, setting new state-of-the-art results on nine language benchmarks”
Children learn general representations with very few explicitly labeled examples, unlike supervised learning approaches like AlexNet that required the enormous manually-labeled ImageNet dataset, suggesting human learning relies on different mechanisms than supervised learning.
“children are able to learn very general representations for concepts like dog with very few explicitly labeled examples”
In 2022, Yann LeCun published 'A Path Towards Autonomous Machine Intelligence,' a 60-page position paper that synthesized Barlow Twins, VICReg, DINO, and other work into a holistic framework for building intelligent machines, arguing world models are the missing piece.
“Following the success of Barlow Twins, VicRag, and Dino V1, in 2022, LeCun brought these and many other threads together into a 60-page position paper called a path towards autonomous machine intelligence.”
OpenAI was founded in 2015 and initially focused their efforts on reinforcement learning, creating OpenAI Gym and Universe, and showing very impressive performance on complex video games.
“OpenAI was founded in 2015 and initially focused their efforts on reinforcement learning, creating OpenAI Gym and Universe, and showing very impressive performance on complex video games.”
GPT-2 (2019) and GPT-3 (2020) were created by aggressively scaling up the GPT-1 approach, with GPT-3 achieving state-of-the-art results across a much broader range of language tasks through sheer scale.
“Other researchers at OpenAI quickly grasped the significance of Radford's results, and the team went all in on this approach, aggressively scaling up to GPT-2 in 2019, GPT-3 in 2020, and ChatGPT in 2022.”
VICReg is a simplified version of Barlow Twins that turned out to be quite effective, and DINO is a similar joint embedding technique developed by colleagues at Meta's Paris lab, demonstrating convergence on joint embedding approaches for vision.
“after that, publish another version, a simplified version basically of bottle twins called VICReg. Which turned out to be quite good. Uh and then simultaneously, another group, some of our colleagues at Fair Paris, were working on uh uh similar methods, which eventually came to be known as DINO. Uh DINO V1, V2, V3. Uh now they have a new version, um which is not called DINO anymore.”
LeCun pioneered convolutional neural networks in the 1980s during expert systems era, and 25 years later AlexNet's breakthrough architecture was uncannily similar to LeCun's 1990s convolutional nets, validating his early insights.
“Yann LeCun saw the revolution coming. In the 1980s, while most of the AI field was busy building expert systems that were explicitly programmed instead of learned from data, Yann pioneered the convolutional neural network. 25 years later, when deep learning began its rise to its now dominant position in AI, the breakthrough deep learning model AlexNet turned out to be uncannily similar to LeCun's convolutional nets from the 1990s.”
Self-supervised learning achieved success in language (next-token prediction) much faster than in vision and other modalities, suggesting language is a particularly rich and informative domain for self-supervised pre-training.
“uh turns out the success of self-supervised learning uh you know, happened in in text and and language way faster than it did in sort of more uh you know, natural uh modalities like like vision.”
DINO outputs an embedding vector for each patch of an image and can perform object segmentation by comparing patch embeddings to reference patches without any segmentation labels, demonstrating the quality of learned representations.
“DINO outputs an embedding vector for each patch of image that it analyzes. If I take this image of myself and take DINO's embedding vector from this image patch on my hand, and compare this embedding vector to the rest of the patches in the image, visualizing how similar each patch is to the hand patch using a color map... Dino does a remarkably good job segmenting my hand from the background.”
Horace Barlow hypothesized in 1961 that neurons in animal and human vision systems operate by reducing redundant information between neurons, a principle later applied by Stefan Deny and colleagues to solve representation collapse in joint embedding architectures.
“Here, Yann is referencing the work of Horace Barlow, who hypothesized in 1961 that the neurons in animal and human vision systems operate by reducing redundant information between neurons. Stefan Deny, a postdoc LeCun was working with in 2020, was familiar with Barlow's work and proposed that one way to avoid representation collapse could be to apply Barlow's idea to the outputs of their networks.”
Self-supervised pre-training on self-supervised learning is the 'bulk of the cake' for intelligence, supervised learning is the 'icing,' and reinforcement learning is the 'cherry'—a hierarchy suggesting reinforcement learning alone is too inefficient to reach human-level intelligence.
“Starting in 2015 or so, I started showing a a slide that has become a bit of a meme in the machine learning community, where I said like, you know, if uh it's the cake slide, right? So, if uh intelligence is a takes the bulk of the cake, it is self-supervised learning, and the icing on the cake is supervised learning, and the cherry on the cake is reinforcement learning. And at the time, people were kind of crazy about reinforcement learning, so I was trying to tell them like, this is not never going to you know, take us to, you know, anywhere close to human-level intelligence uh cuz it's too inefficient.”
Contrastive learning prevents representation collapse in joint embeddings by providing the network both positive examples (views of the same image) and negative examples (different images), but requires large amounts of computation and may need exponentially more negative samples as representation dimensionality increases.
“In LeCun's original Siamese network approach, the team used what's now known as contrastive learning to avoid representation collapse giving the network both positive and negative examples. It turns out that we can apply the same contrastive approach to images and video. Training our network to output similar embeddings for views of the same underlying images or videos and dissimilar embeddings for different images or video. These contrastive methods have been successfully implemented on images and videos, but can run into issues when they're scaled up, requiring large amounts of computation and many negative examples to learn meaningful representations. And LeCun has argued that in the worst case, the number of contrastive samples may grow exponentially with the dimension of the representation.”
DINO V3, released August 2025, achieved 88.4% ImageNet accuracy using joint embedding architecture, marking the first time self-supervised vision models matched state-of-the-art weakly-supervised and supervised models on image classification.
“The DINO V3 paper, released in August 2025, marked an important turning point, achieving a very near state-of-the-art image net accuracy of 88.4% using a joint embedding architecture. As the authors say in their paper, 'All in all, this is the first time that a self-supervised model has reached comparable results to weakly and supervised models on image classification.'”
The Transformer architecture, initially designed for language translation by Google, was repurposed by OpenAI for next-token prediction on self-supervised pre-training, becoming the foundation for GPT and all modern large language models.
“OpenAI was founded in 2015 and initially focused their efforts on reinforcement learning, creating OpenAI Gym and Universe, and showing very impressive performance on complex video games. While much of the company was focused on reinforcement learning, Ilya Sutskever, Alec Radford, and others became interested in a new neural network architecture from Google, the Transformer, initially designed for language translation.”
However, by 2021 fully supervised approaches like Google's Vision Transformer achieved 88.6% ImageNet accuracy, showing self-supervised methods were still inferior despite Barlow Twins' breakthrough.
“However, in the 9 years from the AlexNet paper in 2012 to the Barlow Twins paper in 2021, fully supervised approaches had made significant improvements over AlexNet. In 2020, a team at Google applied the transformer architecture to image classification, achieving a new state-of-the-art ImageNet accuracy of 88.6%.”
Siamese networks, developed by LeCun and colleagues at Bell Labs in the early 1990s for detecting fraudulent signatures, used joint embedding architecture to output similar embedding vectors for authentic signatures and different vectors for fraudulent ones, avoiding the need for generative reconstruction.
“The method Yan is referring to here, Siamese networks, was created by Yan and his collaborators at Bell Labs in the early 1990s when developing systems to detect fraudulent signatures. The system worked by passing a pair of signature images into two copies of the same neural network. The network copies were not trained to generate any kind of data. Instead, they output vectors of numbers, often referred to as embedding vectors.”
Reinforcement learning experienced a renaissance in the mid-2010s with Google DeepMind breakthroughs on Atari games and Go, but has since shown limited applicability and efficiency compared to self-supervised learning.
“Reinforcement learning, where models learn from interacting with their environments instead of from labeled data, experienced a mini renaissance in the mid-2010s, highlighted by Google DeepMind's breakthrough performance on Atari games and the highly complex board game Go.”
A teenager can learn to drive in approximately 20 hours of practice, yet despite having millions of hours of training data, AI systems can only reach Tesla's Level 2 autonomy and nowhere near Levels 3-5, suggesting current approaches are missing a fundamental capability.
“a teenager that can learn to drive a car in around 20 hours of practice. How is it that we have those millions of hours of training data where we have we can train kind of level two system with it, which is what Tesla's doing basically. Um but nowhere near level three, four, five. Okay? Uh yet a 17-year-old can learn to drive in a few hours of practice. Like how does that happen, right?”
The key insight that solved generative model limitations is that during pre-training, what actually matters is not that models are generative, but rather the internal representations and features they learn to solve the prediction task, which are then adapted for downstream tasks.
“In our GPT example, during the crucial pre-training phase, it really doesn't matter that our model is generative. After pre-training on next token prediction, we're left with a model that's essentially a really good auto complete. But this is not the point. What actually matters are the internal representations and features that the model learns to solve the next token prediction task. These learned internal representations are what allows pre-trained models to be quickly adapted into powerful AI assistants.”
Barlow Twins worked surprisingly well, avoiding representation collapse while learning powerful internal representations of images, achieving an ImageNet accuracy of 73.2% on a frozen encoder with a linear probe, outperforming the original fully supervised AlexNet model by over 10 percentage points.
“The Nee and LeCun and their collaborators designed a new loss function for their joint embedding architecture that measured the deviation of their cross-correlation matrix from the identity matrix. Their new method, which they called Barlow Twins, worked surprisingly well, avoiding representation collapse while learning a powerful internal representation of the images that it was trained on.”
If you really want to build reliable agentic systems, they absolutely have to be able to predict the consequences of their actions, so that they can plan a sequence of actions to fulfill the task they are being asked to fulfill and perhaps guarantee safety guardrails.
“So, uh if you really want to build reliable agentic systems, they absolutely have to be able to predict the consequences of their actions. So that you can plan a sequence of actions to do something, first of all to uh fulfill the task that they are being asked to fulfill, but also uh perhaps to, you know, guarantee some safety guardrails, right?”
For full HD video, each pixel can take on 256 discrete values, and there are 1920×1080×3 color pixels, meaning there are something like 10^(15 million) possible next video frames, dwarfing the number of atoms in the observable universe.
“For full HD video in the most general case, each pixel can take on 256 discrete values, and we have 1920 * 1080 * 3 color pixels, meaning there are something like 10 to the power of 15 million possible next video frames, dwarfing the number of atoms in the observable universe.”
Yann has raised a billion dollars to pursue world models and JEPA approaches as an alternative to large language models.
“AI legend Yann LeCun has raised a billion dollars to pursue an alternative approach to AI. Unlike large language models, LeCun's approach is not rooted in language and is not generative.”
By 2017-18, the best systems for learning representations of images are systems that are not generative and do not reconstruct, but instead take an image through an encoder and coerce the encoder to extract as much information as possible with certain properties, for instance by taking two images of the same scene or images corrupted/transformed in different ways and telling the system the representations should be the same.
“Simultaneously, we started realizing in the you know, round 2017-18 that uh the the best system to learn representations of images are systems that do not are not generative. They don't reconstruct. They they you know, you you you get an image and you run it to an encoder and then you try to kind of coerce this encoder to extract as much information as possible with certain properties. So, for example, you take two images of the same scene or you take an image and you corrupt it or transform it in some ways. You run them both through encoders and you tell the system the representation, whatever you extract, should really be the same for those two images”
Building a model that gives the state of the world at time T+1 as a function of state at time T and an action, then using optimization to find an optimal sequence of actions to reach a particular outcome, is classical optimal control going back to the late 1950s Soviet Union and early 1960s West, but what is not classical is learning the model via machine learning.
“You build a model that gives you the state of the world at time T plus one as a function of state of the world at time T, and an action you imagine taking or intervention or control, right? And then if you have this, you can uh predict the outcome of a sequence of actions, and you can by optimization, you can figure out an optimal sequence of actions to arrive at a particular um outcome, right? This is classical optimal control. This is, you know, this is going back to the late '50s in the Soviet Union, early '60s in the in the West. Uh very classical stuff. Yeah. What is not classical is you learn the model. Sure, yeah. a machine learning to learn the model. Right, yeah. What is even less classical is you learn a representation of the input that computes a state, an abstract state representation, and you learn the you know, the the model in that uh in that state. And that's Jepa.”
Common sense in humans can be understood as a collection of world models that allow agents to predict what is likely, plausible, impossible, and to reason, plan, explore, and imagine solutions to problems.
“common sense can be seen as a collection of models of the world that can tell an agent what is likely, what is plausible, and what is impossible. Using such world models, animals can learn new skills with very few trials. They can predict the consequences of their actions. They can reason, plan, explore, and imagine new solutions to problems.”
LLMs cannot predict the consequences of their actions beforehand; they just take the action and then face whatever comes after (as the famous French kings said, 'après moi le déluge' - after me, the flood).
“They just take the action and then apres moi le deluge as you know, as some famous French kings said.”
ChatGPT released in 2022 represented the culmination of aggressive scaling of the GPT approach, but the underlying architecture and pre-training method remained essentially unchanged from GPT-1.
“Other researchers at OpenAI quickly grasped the significance of Radford's results, and the team went all in on this approach, aggressively scaling up to GPT-2 in 2019, GPT-3 in 2020, and ChatGPT in 2022.”
In part two of the video series, the creators will explore V-JEPA-2 implementations to see exactly how these models stack up against LLM-driven approaches.
“In part two, we'll dive deeper into V-Jepa-2 to get a sense for what's really happening inside the model's embedding space. And see how V-JEPA 2 fares as a robotics control algorithm against the rapidly advancing VLA approaches.”