Distillation-based methods for representation learning use two encoders where one is an exponential moving average of the other, forcing the slower encoder to change weights more gradually; this prevents encoder collapse without explicit contrastive samples, and while there is some theoretical work on why this works, the mechanism is somewhat mysterious and LeCun is personally uncomfortable with it, though it empirically works well when implemented carefully.

factualpending

Speaker

Yann LeCun

Evidence Quote

“the encoder on the right...obtains a version of the weights...through exponential moving average...for some reason that prevents collapse...it's a little bit mysterious why this works”

Source

The Shape of AI to Come! Yann LeCun at AI Action Summit 2025— DSAI by Dr. Osbert Tay
Created: 8/11/2026, 1:17:36 AM

Your notes

Loading notes...