factual
Pre-training scaling plateau after synthetic data limits
All major labs have hit a plateau in pre-training scaling over the last two years, because the world's human-generated text data has been largely tokenized and models trained on synthetic data are not enabling scaling to continue.
factualpending
Speaker
ChetanEvidence Quote
“The synthetic data as generated by the LLMs themselves are not enabling the scaling in pre-training to continue.”
Created: 11/18/2026, 6:42:24 AM
Your notes
Loading notes...