Model distillation works through four steps: (1) feeding data to a teacher model (GPT-4) to obtain probability distributions; (2) the teacher generating 'soft labels' with probability outputs (e.g., 85% chance the subject is a cat, 15% chance it's a mat); (3) DeepSeek recording the teacher's thought process including attention scores and hidden states; and (4) DeepSeek learning directly from these soft labels rather than starting from scratch, drastically reducing training time and computation.

definitionpending

Speaker

Dong Sean

Evidence Quote

“step one is feeding data to GPT the teacher model deep seek inputs the sentence into GPT... step two gbt generates a soft labels... step three is recording the teacher's thought process... finally in step four after gaining the teacher's insights deep seek doesn't need to start from scratch”

Source

The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap— Digging into China
Created: 8/11/2026, 7:46:18 AM

Your notes

Loading notes...