Information content regularization methods prevent encoder collapse by maintaining information in the encoder outputs: one approach is to compute the covariance matrix of representation vectors across a batch and try to make it equal to the identity matrix, which forces each variable to be individually informative (high variance) and decorrelated with others.
factualpending
Speaker
Yann LeCunEvidence Quote
“compute the product of that matrix by its transposed...try to make that covariance Matrix equal to Identity”
Created: 8/11/2026, 1:17:36 AM
Your notes
Loading notes...