definition

Pre-training produces calibrated next-token predictors

Pre-training trains a model to imitate all internet content by maximizing log probability of the next token, which forces it to assign probabilities to everything and ends up making the model very calibrated, capable of both generating and assigning probabilities to all kinds of web content and personas.

definitionpending

Speaker

John Schulman (OpenAI Cofounder)

Evidence Quote

“we're training to maximize log probability—it ends up being very calibrated. Not only can it generate all of the content of the web, it can also assign probabilities to everything.”

Source

John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:30 AM

Your notes

Loading notes...