causal
Pre-training builds common sense, post-training builds products
Without good post-training (supervised fine-tuning plus RLHF) you cannot have a good product, but without good pre-training there is not enough common sense for post-training to have any effect — you can only teach a generally intelligent person a lot of skills — so the same RLHF on a bigger pre-trained model (GPT-4 vs 3.5) yields a much better product, and a lot more breakthroughs will come from post-training plus the data flywheel.
causalpending
Speaker
Aravind SrinivasEvidence Quote
“that’s what allows you to build products that users can interact with, collect more data, create a flywheel”
Created: 6/13/2026, 3:26:05 AM
Your notes
Loading notes...