causal
LLM similarity stems from shared pre-training
Different LLMs from different companies are strikingly similar to each other because pre-training on largely overlapping data makes the pre-trained models essentially the same; differentiation only begins to emerge in RL and post-training, where companies devise different training.
causalpending
Speaker
Ilya SutskeverEvidence Quote
“All the pre-trained models are pretty much the same because they pre-train on the same data.”
Source
Ilya Sutskever – We're moving from the age of scaling to the age of research— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:24:31 AM
Your notes
Loading notes...