Pack-Bayes bound
A classical generalization theory that bounds test loss as training loss plus a compression term, explaining why larger models generalize better when they find more compressible solutions despite having higher capacity.
As defined in Inference, Diffusion, World Models, and More | YC Paper Club
Where it comes up · 1
Most important claims · 1 of 1Search all claims for “Pack-Bayes bound” →
- The mystery of overparameterization is resolved by Pack-Bayes theory: as model parameters increase, empirical risk (training loss) decreases because models fit data better, and simultaneously, more compressible solutions are found, so both terms of the Pack-Bayes bound decrease.Inference, Diffusion, World Models, and More | YC Paper Club