Model distillation works through four steps: (1) feeding data to a teacher model (GPT-4) to obtain probability distributions; (2) the teacher generating 'soft labels' with probability outputs (e.g., 85% chance the subject is a cat, 15% chance it's a mat); (3) DeepSeek recording the teacher's thought process including attention scores and hidden states; and (4) DeepSeek learning directly from these soft labels rather than starting from scratch, drastically reducing training time and computation.
definitionpending
Speaker
Dong SeanEvidence Quote
“step one is feeding data to GPT the teacher model deep seek inputs the sentence into GPT... step two gbt generates a soft labels... step three is recording the teacher's thought process... finally in step four after gaining the teacher's insights deep seek doesn't need to start from scratch”
Created: 8/11/2026, 7:46:18 AM
Your notes
Loading notes...