Distillation of an 8-model ensemble (2.4B total parameters) into a single 300M-parameter dense model retains 83% of the loss improvement, allowing data-efficient training to be amortized into a small inference model.
factualpending
Speaker
Ku WooEvidence Quote
“you can take an eight ensemble which is roughly 2.4 billion total parameters and you can distill it into a single dense 300 million parameter model... retaining roughly 83% of the loss improvement”
Created: 8/12/2026, 6:25:34 PM
Your notes
Loading notes...