definition
On-Policy Self-Distillation Distills Sessions Into Weights
On-policy self-distillation (OPSD) distills a session into weights by encouraging the base model to make the same predictions on real-world problems that a model with the full accumulated session context would make, so the base model learns what the context-equipped teacher knows.
definitionpending
Speaker
Dwarkesh PatelEvidence Quote
“we encourage the base model to make the same predictions when trying to solve some real world problem as the model with all the context accumulated after a long session would have made.”
Source
Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job— Dwarkesh Patel PodcastCreated: 11/17/2026, 6:25:50 AM
Your notes
Loading notes...