Self-supervised learning
Learning from raw unlabeled data by predicting parts of the input from other parts (e.g., next-token prediction).
As defined in Aravind Srinivas
Other definitions
Machine learning approach where a system learns representations without explicit labels by solving pretext tasks like predicting masked data from unmasked data
Metas AI Boss Says He DONE With LLMS...Machine learning approach where models learn from unlabeled data by creating labels automatically from the data structure itself (e.g., predicting missing parts, predicting future frames); does not require human annotations
Yann LeCun: World Models: Enabling the next AI revolution
Where it comes up · 10
- Yann LeCun: World Models: Enabling the next AI revolution5 claims
- Research Pioneer5 claims
- AI: Grappling with a New Kind of Intelligence | World Science Festival3 claims
- Yann LeCun's $1B Bet Against LLMs [Part 1]3 claims
- Metas AI Boss Says He DONE With LLMS...2 claims
- The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap2 claims
- Yann LeCun: Special Lecture on AI and World Models2 claims
- Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI1 claim
and 2 more
Most important claims · 10 of 25Search all claims for “Self-supervised learning” →
- A system trained with self-supervised learning on images needs only 2-3 examples of a new object (like an elephant) to recognize it, making it more efficient than supervised learning which requires thousands of labeled examples—this is closer to how humans learn.AI: Grappling with a New Kind of Intelligence | World Science Festival
- Yann LeCun was right in 2016 — controversially at the time when RL was hot — that RL is just the cherry on the cake, supervised learning is the icing, and the bulk of intelligence is in the unsupervised (self-supervised) cake; this turned out to be the recipe for ChatGPT, where most compute goes to next-token pre-training, then supervised fine-tuning, then RLHF for conversational ability.Aravind Srinivas
- Babies and animals learn primarily through observation and self-supervised learning, acquiring basic concepts like object permanence at 9 months, intuitive physics (gravity, inertia) by age 10 months, and the difference between animate and inanimate objects without explicit instruction.Yann LeCun: Special Lecture on AI and World Models
- Joint-Embedding Predictive Architecture (JEPA) trains by encoding both a full input and a corrupted version, then predicting the representation of the full input from the corrupted one in abstract representation space, rather than reconstructing pixels; this lets the encoder discard unpredictable noise and preserve only predictable, modelable structure.Research Pioneer
- Self-supervised learning achieved success in language (next-token prediction) much faster than in vision and other modalities, suggesting language is a particularly rich and informative domain for self-supervised pre-training.Yann LeCun's $1B Bet Against LLMs [Part 1]
- Supervised learning requires large labeled datasets where a human tells the system what each input corresponds to, making it impractical for obscure languages, unwritten languages, or specialized domains where labeled data is scarce; this limitation motivated the development of self-supervised learning.AI: Grappling with a New Kind of Intelligence | World Science Festival
- Self-supervised learning works by showing models text with missing words, removing words, and training the system to predict the missing words, which forces the network to learn representations that capture meaning, syntax, grammar, and internal structure of language that can then be applied to downstream tasks.AI: Grappling with a New Kind of Intelligence | World Science Festival
- Self-supervised prediction algorithms learn more structure when input data is more redundant; perceptual/sensory inputs like vision have far more redundancy than text (which is already compressed and less redundant), so self-supervision works better on perception than on language.Research Pioneer
- Meta has built speech-to-speech translation systems that work directly through an internal representation of discrete speech units (Textless NLP) without going through text, enabling translation even for languages that have no written form.Research Pioneer
- Self-supervised pre-training on self-supervised learning is the 'bulk of the cake' for intelligence, supervised learning is the 'icing,' and reinforcement learning is the 'cherry'—a hierarchy suggesting reinforcement learning alone is too inefficient to reach human-level intelligence.Yann LeCun's $1B Bet Against LLMs [Part 1]