soft labels
In model distillation, probability distributions generated by a teacher model that show the likelihood of different possible answers rather than just one correct answer; for example, '85% cat, 15% mat' instead of simply 'cat', providing richer learning signal
As defined in The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap
Where it comes up · 2
- The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap5 claims
- Model Distillation: How to Steal a Billion-Dollar AI2 claims
Most important claims · 7 of 7Search all claims for “soft labels” →
- Dark knowledge—the full probability distribution a teacher model assigns across all possible next tokens—is the signal that makes distillation work; soft labels reveal how a model ranks alternatives (e.g., 90% cat, 9% dog, 1% car from a photo), not just the final answer.Model Distillation: How to Steal a Billion-Dollar AI
- In black box distillation, only text outputs are visible; prompts are sent, answers are saved, and small models are trained on those outputs alone, which is a weaker signal than soft labels but repeated across 100,000 examples burns the teacher's habits into the student.Model Distillation: How to Steal a Billion-Dollar AI
- Model distillation works through four steps: (1) feeding data to a teacher model (GPT-4) to obtain probability distributions; (2) the teacher generating 'soft labels' with probability outputs (e.g., 85% chance the subject is a cat, 15% chance it's a mat); (3) DeepSeek recording the teacher's thought process including attention scores and hidden states; and (4) DeepSeek learning directly from these soft labels rather than starting from scratch, drastically reducing training time and computation.The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap
- In model distillation step two, GPT generates 'soft labels' after analyzing a sentence—a probability distribution showing, for example, an 85% chance the subject is a cat and 15% chance it's a mat, which not only tells DeepSeek the right answer but provides insight into how GPT reasoned to that conclusion.The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap
- In model distillation step four, after gaining the teacher's insights, DeepSeek does not need to start from scratch—it directly learns from the teacher's experience by repeatedly learning from soft labels, drastically reducing training time, saving computation power, and enabling results comparable to GPT-4.The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap
- DeepSeek achieved GPT-4-scale model results with only $5.5 million in funding—a fraction of GPT's cost—by using model distillation, where a fully trained GPT model serves as a teacher providing soft labels and insights into its reasoning process.The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap
- DeepSeek accesses GPT soft labels through legal public APIs, similar to how users would access ChatGPT on a website, except at much larger scale—this is not considered stealing as long as they follow terms of use and do not abuse the service, though whether DeepSeek fully complies is unclear.The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap