encoder
A neural network component that transforms a raw input (image, video, text) into a high-level abstract representation.
As defined in Research Pioneer
Other definitions
Neural network component that transforms high-dimensional input data into a lower-dimensional abstract representation or embedding, extracting relevant features while discarding irrelevant details
The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
Where it comes up · 9
- Illustrated Guide to Transformers Neural Network: A step by step explanation14 claims
- The Shape of AI to Come! Yann LeCun at AI Action Summit 202512 claims
- Yann LeCun's $1B Bet Against LLMs [Part 1]3 claims
- LeWorldModel: Stable End-to-End JEPA World Models From Pixels2 claims
- Renishaw: The Precision Pioneers2 claims
- Dolby: The Sound Standard2 claims
- Yann LeCun: Special Lecture on AI and World Models2 claims
- Research Pioneer1 claim
and 1 more
Most important claims · 10 of 39Search all claims for “encoder” →
- JEPA (Joint Embedding Predictive Architecture) avoids the pixel prediction problem by mapping observations to embeddings and training a predictor to forecast future embeddings rather than pixels, allowing the model to focus on salient scene features instead of random unpredictable details.Yann LeCun's $1B Bet Against LLMs [Part 1]
- LayWM is designed with only 15 million parameters, which is a tiny fraction of modern large language models that have hundreds of billions or over a trillion parameters, and can be trained end-to-end entirely from raw pixels on a single consumer GPU in a matter of hours.LeWorldModel: Stable End-to-End JEPA World Models From Pixels
- DINO (a distillation-based method developed by Fair Paris researchers led by Mathilde Caron) and similar approaches learn generic visual features that transfer well to downstream tasks without needing negative samples, and these methods have been extended to video (DINO v2, v3) to learn temporal representations from video segments.The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
- Joint-Embedding Predictive Architecture (JEPA) trains by encoding both a full input and a corrupted version, then predicting the representation of the full input from the corrupted one in abstract representation space, rather than reconstructing pixels; this lets the encoder discard unpredictable noise and preserve only predictable, modelable structure.Research Pioneer
- The decoder's first multi-headed attention layer operates differently from the encoder because the decoder is autoregressive and generates sequences word-by-word, requiring it to prevent attending to future tokens that haven't been generated yet.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Transformers must add positional encoding information to input embeddings because unlike recurrent neural networks, transformers have no recurrence and therefore need explicit position information to understand word order.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Alternative models like DinoWM solved representation collapse by pre-training on 142 million real-world images and freezing the encoder, effectively providing the AI with a massive pre-existing 'cheat sheet' of visual understanding before learning physics.LeWorldModel: Stable End-to-End JEPA World Models From Pixels
- The 'Attention is All You Need' paper introduced a novel attention-based encoder-decoder neural network architecture called Transformers that demonstrated superior performance compared to previous models.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Information content regularization methods prevent encoder collapse by maintaining information in the encoder outputs: one approach is to compute the covariance matrix of representation vectors across a batch and try to make it equal to the identity matrix, which forces each variable to be individually informative (high variance) and decorrelated with others.The Shape of AI to Come! Yann LeCun at AI Action Summit 2025
- Stacking encoder layers allows each layer to learn different attention representations, potentially boosting the predictive power of the transformer network.Illustrated Guide to Transformers Neural Network: A step by step explanation