representation collapse
A failure mode in joint embedding architectures where the network learns a trivial solution that outputs identical embeddings for all inputs, avoiding useful feature learning
As defined in Yann LeCun's $1B Bet Against LLMs [Part 1]
Other definitions
A failure mode in self-supervised learning where the encoder outputs the same constant embedding for all inputs, causing the embedding space to collapse into a single point with no useful information despite low training loss
What Is Yann LeCun Cooking? JEPA Explained Simply
Where it comes up · 3
- LeWorldModel: Stable End-to-End JEPA World Models From Pixels4 claims
- Yann LeCun's $1B Bet Against LLMs [Part 1]4 claims
- What Is Yann LeCun Cooking? JEPA Explained Simply4 claims
Most important claims · 10 of 12Search all claims for “representation collapse” →
- The historical problem with training world models through prediction is 'representation collapse'—where an AI can achieve perfect prediction accuracy by mapping all inputs to the same blank representation (e.g., zero) without learning any actual physics, because the loss function only penalizes prediction error, not learning quality.LeWorldModel: Stable End-to-End JEPA World Models From Pixels
- The core innovation in LayWM's regularization is SigReg (Sketched Isotropic Gaussian Regularizer), which enforces geometric diversity by forcing the AI's latent embeddings to match an isotropic Gaussian distribution—a perfectly diverse multi-dimensional bell curve—thereby mathematically forbidding representation collapse.LeWorldModel: Stable End-to-End JEPA World Models From Pixels
- Representation collapse in joint embedding architectures—where networks trivially output identical embeddings for all inputs—was the primary unsolved problem preventing joint embeddings from learning useful representations until Barlow Twins and related approaches.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Horace Barlow hypothesized in 1961 that neurons in animal and human vision systems operate by reducing redundant information between neurons, a principle later applied by Stefan Deny and colleagues to solve representation collapse in joint embedding architectures.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Alternative models like DinoWM solved representation collapse by pre-training on 142 million real-world images and freezing the encoder, effectively providing the AI with a massive pre-existing 'cheat sheet' of visual understanding before learning physics.LeWorldModel: Stable End-to-End JEPA World Models From Pixels
- Contrastive learning prevents representation collapse in joint embeddings by providing the network both positive examples (views of the same image) and negative examples (different images), but requires large amounts of computation and may need exponentially more negative samples as representation dimensionality increases.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Representation collapse is a failure mode where both the context and target encoders in JEPA can learn to output the same constant embedding for all inputs, with the predictor simply outputting a constant embedding that always matches, resulting in extremely low training loss but learning absolutely nothing about the world.What Is Yann LeCun Cooking? JEPA Explained Simply
- Older models like PLDM attempted to prevent representation collapse by designing loss functions with seven different interacting terms, creating extreme training instability due to the combination lock problem—adjusting one parameter unpredictably shifts all others.LeWorldModel: Stable End-to-End JEPA World Models From Pixels
- EMA is heuristic-based because stability depends on EMA schedules, weight sharing, and training dynamics, which can be fragile and require manual tuning, motivating research into other methods for preventing representation collapse.What Is Yann LeCun Cooking? JEPA Explained Simply
- The first practical solution researchers used to prevent representation collapse was updating the target encoder with exponential moving average (EMA), where instead of both encoders learning freely, the target encoder is updated very slowly as a delayed version of the context encoder.What Is Yann LeCun Cooking? JEPA Explained Simply