Dark knowledge—the full probability distribution a teacher model assigns across all possible next tokens—is the signal that makes distillation work; soft labels reveal how a model ranks alternatives (e.g., 90% cat, 9% dog, 1% car from a photo), not just the final answer.

definitionpending

Speaker

Unidentified Speaker — Model Distillation: How to Steal a Billion-Dollar AI [gacMZf5rqYA]

Evidence Quote

“When a teacher model answers a question, it does not pick one word. It assigns a probability to every possible next token... Those are called soft labels.”

Source

Model Distillation: How to Steal a Billion-Dollar AI— Kai
Created: 8/12/2026, 2:31:09 AM

Your notes

Loading notes...