causal

Pre-training builds common sense, post-training builds products

Without good post-training (supervised fine-tuning plus RLHF) you cannot have a good product, but without good pre-training there is not enough common sense for post-training to have any effect — you can only teach a generally intelligent person a lot of skills — so the same RLHF on a bigger pre-trained model (GPT-4 vs 3.5) yields a much better product, and a lot more breakthroughs will come from post-training plus the data flywheel.

causalpending

Speaker

Aravind Srinivas

Evidence Quote

“that’s what allows you to build products that users can interact with, collect more data, create a flywheel”

Source

Aravind Srinivas— Lex Fridman Podcast
Created: 6/13/2026, 3:26:05 AM

Your notes

Loading notes...