Distillation-based methods for representation learning use two encoders where one is an exponential moving average of the other, forcing the slower encoder to change weights more gradually; this prevents encoder collapse without explicit contrastive samples, and while there is some theoretical work on why this works, the mechanism is somewhat mysterious and LeCun is personally uncomfortable with it, though it empirically works well when implemented carefully.
factualpending
Speaker
Yann LeCunEvidence Quote
“the encoder on the right...obtains a version of the weights...through exponential moving average...for some reason that prevents collapse...it's a little bit mysterious why this works”
Created: 8/11/2026, 1:17:36 AM
Your notes
Loading notes...