definition

On-Policy Self-Distillation Distills Sessions Into Weights

On-policy self-distillation (OPSD) distills a session into weights by encouraging the base model to make the same predictions on real-world problems that a model with the full accumulated session context would make, so the base model learns what the context-equipped teacher knows.

definitionpending

Speaker

Dwarkesh Patel

Evidence Quote

“we encourage the base model to make the same predictions when trying to solve some real world problem as the model with all the context accumulated after a long session would have made.”

Source

Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:25:50 AM

Your notes

Loading notes...