Distillation of an 8-model ensemble (2.4B total parameters) into a single 300M-parameter dense model retains 83% of the loss improvement, allowing data-efficient training to be amortized into a small inference model.

factualpending

Speaker

Ku Woo

Evidence Quote

“you can take an eight ensemble which is roughly 2.4 billion total parameters and you can distill it into a single dense 300 million parameter model... retaining roughly 83% of the loss improvement”

Source

Inference, Diffusion, World Models, and More | YC Paper Club— Y Combinator
Created: 8/12/2026, 6:25:34 PM

Your notes

Loading notes...