contrastive methods
Training methods that learn by pushing down the energy of real/positive data samples and pushing up the energy of generated/negative samples; used to prevent collapse in representation learning.
As defined in The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
Other definitions
A class of representation learning techniques that learn by pushing down the energy/loss for data points and pushing up the energy for generated negative samples. Requires explicit generation of negative examples to occupy off-manifold space but scales poorly in high dimensions.
The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
Where it comes up · 3
- The Shape of AI to Come! Yann LeCun at AI Action Summit 20255 claims
- What Is Yann LeCun Cooking? JEPA Explained Simply4 claims
- Yann LeCun: World Models: Enabling the next AI revolution2 claims
Most important claims · 10 of 11Search all claims for “contrastive methods” →
- Training JEPA architectures requires designing cost functions that produce low energy on training data but high energy outside the training manifold; there are only two classes of methods for this: (1) contrastive methods push down energy on real data and push up energy on synthetic negative samples, but they don't scale well to high dimensions and require many negative samples; (2) regularized methods use regularizers to minimize the volume of low-energy space, which scales better.The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
- LeCun's core recommendations for AI research: (1) Abandon generative models in favor of JEPA, (2) Skip probabilistic models in favor of energy-based models, (3) Replace contrastive methods with regularization methods, (4) Abandon reinforcement learning except as a last resort when models or cost functions are inaccurateThe Shape of AI to Come! Yann LeCun at AI Action Summit 2025
- LeCun's practical recommendations for AI research include: abandon generative models, work on JEPAs and representation-space prediction instead, abandon probabilistic models (they are intractable for high-dimensional outputs), use energy-based models, abandon contrastive methods in favor of regularized methods, abandon reinforcement learning as a primary approach (use it only as a last resort when model or cost functions are inaccurate).The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
- Joint-embedding systems can collapse by ignoring inputs and producing constant representations; preventing collapse is the central problem in self-supervised learning for these systems and requires either contrastive methods (pushing up energy of non-data points) or regularized methods (minimizing volume of low-energy regions)Yann LeCun: World Models: Enabling the next AI revolution
- The downside of contrastive methods is they rely heavily on negative samples, requiring a large batch of other images to push away from, meaning you need very large batch sizes or memory banks, which makes training computationally expensive and harder to scale.What Is Yann LeCun Cooking? JEPA Explained Simply
- Techniques like Barlow Twins and VICReg implement dimension contrastive methods by measuring correlation between embedding dimensions and penalizing the model if multiple dimensions start carrying the same signal, avoiding the need for large batches of negative samples.What Is Yann LeCun Cooking? JEPA Explained Simply
- Contrastive methods like SimCLR (published 2020) train models by creating two augmented views of the same image and training the model to recognize they come from the same sample, pulling their embeddings closer together while pushing embeddings of other images farther apart.What Is Yann LeCun Cooking? JEPA Explained Simply
- Training joint embedding architectures requires ensuring that the energy function (divergence measure between predicted and target representations) is low on the training data manifold but high outside it, which can be achieved through two classes of methods: contrastive methods (explicitly pushing negative samples up) and regularized methods (constraining the manifold volume using regularizers like correlation or covariance penalties).The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
- Information can be maximized by ensuring all rows of a representation matrix are different (making each sample distinct) or all columns are different (making each variable independent), with the latter (dimension contrastive methods) being preferable because independent variables are more informativeYann LeCun: World Models: Enabling the next AI revolution
- Despite the progress of dimension contrastive methods, they still relied on multiple loss terms and carefully balanced hyperparameters to keep representation stable, motivating Yann LeCun's latest bet called LeJepa released in November 2025.What Is Yann LeCun Cooking? JEPA Explained Simply