Barlow Twins
Self-supervised learning technique developed by researchers working with LeCun using joint-embedding architecture with information maximization.
As defined in Yann LeCun: World Models: Enabling the next AI revolution
Where it comes up · 3
- Yann LeCun's $1B Bet Against LLMs [Part 1]5 claims
- What Is Yann LeCun Cooking? JEPA Explained Simply2 claims
- Yann LeCun: World Models: Enabling the next AI revolution1 claim
Most important claims · 8 of 8Search all claims for “Barlow Twins” →
- Representation collapse in joint embedding architectures—where networks trivially output identical embeddings for all inputs—was the primary unsolved problem preventing joint embeddings from learning useful representations until Barlow Twins and related approaches.Yann LeCun's $1B Bet Against LLMs [Part 1]
- In 2022, Yann LeCun published 'A Path Towards Autonomous Machine Intelligence,' a 60-page position paper that synthesized Barlow Twins, VICReg, DINO, and other work into a holistic framework for building intelligent machines, arguing world models are the missing piece.Yann LeCun's $1B Bet Against LLMs [Part 1]
- VICReg is a simplified version of Barlow Twins that turned out to be quite effective, and DINO is a similar joint embedding technique developed by colleagues at Meta's Paris lab, demonstrating convergence on joint embedding approaches for vision.Yann LeCun's $1B Bet Against LLMs [Part 1]
- LeJepa can train JEPA-style models without relying on EMA while still avoiding collapse and learning strong representations, achieving competitive or better performance compared to VICReg, Barlow Twins, and SimCLR on standard vision benchmarks, with accuracy levels similar to state-of-the-art DINO on ImageNet.What Is Yann LeCun Cooking? JEPA Explained Simply
- However, by 2021 fully supervised approaches like Google's Vision Transformer achieved 88.6% ImageNet accuracy, showing self-supervised methods were still inferior despite Barlow Twins' breakthrough.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Barlow Twins worked surprisingly well, avoiding representation collapse while learning powerful internal representations of images, achieving an ImageNet accuracy of 73.2% on a frozen encoder with a linear probe, outperforming the original fully supervised AlexNet model by over 10 percentage points.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Techniques like Barlow Twins and VICReg implement dimension contrastive methods by measuring correlation between embedding dimensions and penalizing the model if multiple dimensions start carrying the same signal, avoiding the need for large batches of negative samples.What Is Yann LeCun Cooking? JEPA Explained Simply
- Techniques like MNCR, NCR squared, WMSE, Sigreg, VICReg, and Barlow Twins all implement information maximization for joint-embedding systems, with JEPA gaining popularity (1,700 papers mention it on Google Scholar)Yann LeCun: World Models: Enabling the next AI revolution