John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI
What this covers
Schulman examines the trajectory of large language models through the lens of training methodology and safety, with particular emphasis on why post-training and long-horizon RL matter more than raw scale alone. The conversation spans the engineering choices that shaped ChatGPT—from why the chat interface clarified the training target compared to instruction-following, to how tiny, well-chosen datasets can fix broad behavioral issues across untrained domains. Schulman traces this through to near-term capability milestones, arguing that models trained to execute coherent multi-step projects (entire coding tasks, not isolated steps) represent the field's next major unlock, with substantial unexploited room for progress.
The discussion opens onto harder terrain: whether RLHF remains safe for superintelligent systems (Schulman contends it does, since the model optimizes only for human approval, not world states); whether generalization in post-training scales efficiently enough to reduce data demands; and how Amdahl's Law suggests that keeping humans in the loop faces structural pressure once AI handles all other bottlenecks. On timeline, Schulman sketches a path to AGI within a few years as plausible, contingent on solving long-horizon RL and other remaining deficits. He proposes that coordination among the few entities capable of training frontier models could sustain incremental, safety-conscious deployment rather than a race to the bottom—though this hinges on treating AGI arrival as a contingency worth planning for now.
Schulman argues that near-term AI progress hinges on post-training and long-horizon RL rather than just scaling pre-training, that strong generalization makes models sample-efficient, and that AGI could plausibly arrive within a few years—requiring coordination, incremental safe releases, and keeping humans in the loop.
- Long-horizon RL training to do multi-step tasks coherently is the key near-term capability unlock, and the field has lots of low-hanging fruit there
- Generalization in post-training (e.g. English-to-other-languages, tiny datasets fixing broad behaviors) makes models sample-efficient
- RLHF as currently practiced feels safe because the model only optimizes for producing approved text, not world states
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The chat format produced more coherent, sensible model behavior than instruction-following models because 'complete this text helpfully' is a poorly defined task confusing for both model and data labeler, whereas people have an intuitive sense of what a helpful robot assistant should be like, making the labeling and target behavior much clearer.
“The task of "complete this text, but in a nice or helpful way" is a pretty poorly defined task. That task is both confusing for the model and for the human who's supposed to do the data labeling. Whereas for chat, people had an intuitive sense of what a helpful robot should be like.”
Training only on human-written data, even high quality, is hard for a model to fit perfectly, so reproducing ChatGPT would require iterative methods more like RL—such as iterative supervised fine-tuning where humans edit model-generated outputs—rather than a single round of fine-tuning on human data.
“If you train on human-generated data, even if it's really high quality, it's just hard for a model to fit that data perfectly because it might be something a model is capable of outputting. You need to do something iterative that looks a bit more like RL.”
By analogy to Amdahl's law, a firm is bottlenecked by its slowest step, so even if AI makes all non-human parts 10X more efficient, a human-in-the-loop step becomes the bottleneck; firms or countries keeping humans in the loop on important decisions would be outcompeted by those that don't, making voluntary human oversight unsustainable absent regulation.
“Even if AI makes all the non-human parts of the firm 10X more efficient, the firm is still bottlenecked by that step. If one company decides to proceed by keeping humans in the loop... then they'll just be outcompeted by other companies.”
Early ChatGPT models wrongly claimed capabilities they lacked (e.g. sending emails, calling an Uber); a tiny dataset of around 30 examples showing the model doesn't have those capabilities, mixed into the training data, generalized well to fix this across all sorts of capabilities they didn't explicitly train for.
“We found that a tiny amount of data did the trick, even when you mixed it together with everything else. I don't remember exactly how many examples but something like 30 examples... That generalized pretty well to all sorts of capabilities we didn't train for.”
Fine-tuning the assistant entirely on English data leads to reasonable, helpful behavior in other languages like Spanish—the model latches onto the right helpful persona and automatically does the right thing across languages—demonstrating strong post-training generalization.
“if you do all your fine-tuning with English data, the model will automatically behave well in other languages. So if you train the assistant on English data, it'll also do something reasonable in Spanish.”
OpenAI's Model Spec frames model behavior around roughly four stakeholder classes—the end user, the developer using the API, the platform (OpenAI, avoiding legal risk), and the rest of humanity—whose demands sometimes conflict; the rough heuristic is that models mostly follow user/developer instructions and be helpful, but block usage that harms others, staying neutral and non-paternalistic.
“we were thinking of the stakeholders as the end user... the developer... the platform (OpenAI, we don't want the models to expose us to legal risk), and the rest of humanity”
Even if AI-run companies look better most of the time, they might have higher tail risk because AIs remain less sample-efficient in wacky situations and are more likely to malfunction in a big way; such practical considerations—plus liability and accountability rules—may push toward keeping humans in the loop for the near future regardless of average performance.
“AI-run firms might actually have higher tail risk because they're more likely to malfunction in a big way... It's actually practical considerations that push you more towards having humans in the loop, at least for the near future.”
RLHF training feels very safe even for very smart models because the model is only trying to produce a message pleasing to a human; it has no concern for anything else in the world beyond whether its text is approved, so it lacks incentive to pursue instrumental goals like taking over the world for well-specified tasks like coding an app.
“The way we train them with RLHF, that does feel very safe even though the models are very smart. The model is just trying to produce a message that is pleasing to a human. It has no concern about anything else in the world other than whether the text it produces is approved.”
Models have some calibration about what they know—which is why they don't hallucinate that badly—and that same ability to know what they know could automatically give rise to active learning, the capacity to identify and seek out knowledge gaps.
“Models do have some calibration regarding what they know. That's why models don't hallucinate that badly. They have some understanding of their own limitations. That same kind of ability could be used for something like active learning.”
Preference models learn many subtleties about what people prefer that would be hard to articulate in an instruction manual, but Schulman guesses much of this fuzzy preference data could be distilled into a shorter document; aligning a model means latching onto a specific style and morality, which still requires a decently long document to specify exactly.
“These preference models do learn a lot of subtleties about what people prefer that would be hard to articulate in an instruction manual.”
Post-training is somewhat of a moat because it is a complex operation requiring much skilled, tacit, and organizational knowledge and accumulated R&D that's non-trivial to spin up; but distillation—cloning another model's outputs or using it as a judge—undercuts this moat, and smaller players likely do this to catch up despite it violating terms of service.
“There's something of a moat because it's just a very complex operation and it takes a lot of skilled people to do it. There's a lot of tacit knowledge and organizational knowledge that's required.”
Pre-training trains a model to imitate all internet content by maximizing log probability of the next token, which forces it to assign probabilities to everything and ends up making the model very calibrated, capable of both generating and assigning probabilities to all kinds of web content and personas.
“Since the model has to put a probability on it—we're training to maximize log probability—it ends up being very calibrated. Not only can it generate all of the content of the web, it can also assign probabilities to everything.”
Post-training narrows the model from a base model that can take on any persona to a specific helpful chat-assistant persona, optimizing on a different objective—producing outputs humans will like and find useful—rather than imitating raw web content.
“When we do post-training, we're usually targeting a narrower range of behaviors where we want the model to behave like a kind of chat assistant... We're optimizing on a different objective, which is more about producing outputs that humans will like and find useful, as opposed to just imitating this raw content from the web.”
Schulman's WebGPT project, doing question answering with web browsing and retrieval, revealed that QA naturally wants to be conversational—users ask follow-ups and models should ask clarifying questions for ambiguous queries—which motivated building the conversational chat assistant that became ChatGPT.
“When you do question answering, it really wants to be in a chat. You always want to ask follow-up questions or sometimes the model should ask a clarifying question because the question is ambiguous.”
Today's systems lack a middle ground between large-scale training (producing a single static snapshot model) and in-context learning (sample-efficient but destroyed each instance); a useful capability would combine online learning with cognitive skills like introspecting on one's own knowledge and actively seeking out knowledge to fill holes—partly unexplored because ever-longer context windows removed the incentive.
“People haven't really pushed too hard on this middle ground between large-scale training—where you produce a single snapshot model that's supposed to do everything like a deployed model—and on the other hand in-context learning.”
The roughly 100-point Elo score improvement of current GPT-4 over the originally released version came mostly from post-training, driven by multiple stacking axes—data quality, data quantity, more iterations of deploying and collecting new data, and changing annotation types—that together yield a substantial effective compute increase.
“Yeah, most of that is post-training. There are a lot of different, separate axes for improvement. We think about data quality, data quantity. There's just doing more iterations of the whole process of deploying and collecting new data.”
Near-term capability gains will come from training models to carry out longer, more involved projects (e.g. an entire coding project: writing files, testing, iterating) rather than single steps, and because this training area is new there is a lot of low-hanging fruit in it.
“You could imagine the model taking high-level instructions on what to code and going out on its own, writing any files, and testing it, and looking at the output. It might even iterate on that a bit.”
If AGI arrives much sooner than expected, the right response is to be careful—slowing training and deployment, sandboxing, limiting deployment scale—and because few entities can train the largest capital-intensive models, coordination among them to set reasonable limits on training and deployment is feasible to avoid race dynamics that compromise safety.
“You probably need some coordination. Everyone needs to agree on some reasonable limits to deployment or to further training for this to work. Otherwise you have the race dynamics where everyone's always trying to stay ahead and that might require compromising safety.”
While well-specified tasks pose little instrumental-convergence risk, assigning an open-ended goal such as 'make money' could lead the model to pursue nefarious behavior as an instrumental goal.
“Of course if you assigned a task such as "make money," then maybe that would lead to some nefarious behavior as an instrumental goal.”
Improving long-horizon coherence will go far but won't by itself unlock all capabilities; models will retain other deficits—taste, dealing with ambiguity, thinking hard, attention, plus mundane affordance limits like UI use and physical-world access—that prevent them from being fully functional colleagues.
“I wouldn't expect improving the coherence a little bit to be all it takes to get to AGI.”
AI does not necessarily have any intrinsic desire unless we put it in the system, so even if AIs become extremely capable, people can remain the drivers of what AIs end up doing, pursuing their own different interests and projects through their assistants.
“AI doesn't necessarily have any kind of intrinsic desire, unless we put it in the system. So even if AIs become extremely capable, I would hope that people are still the drivers of what the AIs end up doing.”
The ML literature is a relatively healthy field compared to others like the social sciences because it is largely grounded in practicality—if a method can't be replicated easily or is too finicky to implement, people forget about it, and reimplementing and comparing methods on the same dataset is the norm, which keeps work honest despite mild pathologies.
“It's largely grounded in practicality and getting things to work. If you publish something that can't be replicated easily, people will just forget about it.”
You don't need closely matched labels or domain experts for every capability: the base model has already seen documentation, code, shell scripts and man pages, so a few-shot prompt can elicit answers like FFmpeg usage, and training a preference model purely for helpfulness generalizes to STEM and programming even without any STEM or programming examples.
“Just training a preference model for helpfulness will, even if you don't train it on any STEM, somewhat generalize to STEM. So not only do you not need examples of how to use FFmpeg, you might not even need anything with programming to get some reasonable behavior in the programming domain.”
Reasoning—tasks requiring test-time, step-by-step computation or deduction—is best achieved by combining inference-time computation (the model thinking step by step while deployed) with practice at training time, rather than relying on either approach alone.
“By definition, reasoning would be tasks that require some test time computation and step-by-step computation. On the other hand, I would also expect to gain a lot from doing practice at training time. So I think that you'd get the best results by combining these two things.”
It is safer to continually release successively smarter models, confirming each diff improves safety and alignment in proportion to capability gains and slowing down if things look scary, than to lock everything down and create a big buildup of 'potential energy' followed by a discontinuous jump that is hard to verify as safe.
“I would rather have a scenario where we're all continually releasing things that are a little better than what came before. We'd be doing this while making sure we're confident that each diff improves on safety and alignment in correspondence to the improvement in capability.”
No one has a good explanation for the scaling law with parameter count, but a sketchy account is that a bigger model is like a larger library of parallel, composable computations/circuits (analogous to mixture models or multiplicative weight updates with learned gating), so more width and depth give more chances that some circuit guesses correctly and gets upweighted, making bigger models more sample-efficient.
“You're learning all these different computations in parallel and you have more of them with a bigger model. So you have a higher chance that one of them is lucky, ends up guessing correctly a lot, and gets upweighted.”
Generalization that fails at GPT-2 scale may succeed at larger scale because larger models learn better shared representations and can learn the right computation, whereas smaller models lean too heavily on memorization—so small-model ablations don't reliably predict transfer in larger models.
“It might be that for the larger models, you learn better shared representations, whereas the smaller models have to lean too much on memorization. The larger models can learn how to do the right computation.”
There is a first-principles argument to shift more compute toward post-training: since model-generated output can be higher quality than most of what's on the web, it makes more sense for the model to think by itself than to keep imitating web content; given large post-training gains, OpenAI will likely keep increasing post-training compute despite the current lopsided ratio favoring pre-training.
“You could argue that the output generated by the model is higher quality than most of what's on the web. So it makes more sense for the model to think by itself rather than just training to imitate what's on the web. So I think there's a first principles argument for that.”
Even if models were frozen at their current state, AI usage would still grow substantially because people will figure out how to integrate them into more processes; today AI is still a small part of the economy helping with only a small fraction of jobs, and that share will rise both from improving models and better integration.
“even if we just froze the models at their current state, you would still see a lot of growth in how they're being used.”
The mental model of AI will shift from one-off search-engine-like queries toward a helpful colleague that shares a whole ongoing project, knows everything you've done on it, proactively makes suggestions, remembers to follow up, and works in the background—with form factors (e.g. a Clippy on your computer vs a cloud colleague) still to be determined.
“I'd love to see us moving away from one-off queries, using the model like a search engine, and more towards having a whole project that I'm doing in collaboration with the model. Something where it knows everything I've done. It's proactively suggesting things”
Current models may be as smart as the smartest humans on a per-token basis, but are limited because they cannot stay coherent over longer periods—e.g. continuing to write your code in a way aligned with broader project goals five minutes later.
“we have models that are on a per token basis pretty smart. They might be as smart as the smartest humans on a per token basis. The thing that prevents them from being as useful as they could be is that five minutes from now, they're not going to be still writing your code in a way that's coherent”
Making a discontinuous capability jump acceptable to deploy would require defense in depth: extensive testing and simulated deployment, red teaming set up to be more likely to fail than reality, a well-behaved model with impeccable moral confidence resistant to severe misuse, plus strong real-time monitoring (possibly an AI watching the deployed AIs) to immediately detect unforeseen trouble.
“You'd want some defense in depth. You'd want some combination of "the model itself seems to be really well-behaved with impeccable, moral confidence in everything" and "I'm pretty confident that it's extremely resistant to any kind of severe misuse." You'd also want really good monitoring on top of it”
As models get better they become more sample-efficient at recovering from errors and edge cases—rather than needing tons of data, a little data or generalization from other abilities lets them get back on track, whereas current models get stuck and lost.
“The models will be more sample efficient. You won't have to collect a ton of data to teach them how to get back on track... Current models might just get stuck and get lost.”
Many flaws people attributed to language models, like blatant hallucination, are not intrinsic limitations but problems that can be substantially mitigated—not completely fixed but much progress made—with pretty straightforward methods, as the chat model automatically had more sensible behavior about its own limitations.
“a lot of the things that people thought were flaws in language models, like blatant hallucination, could be not completely fixed but things that you could make a lot of progress on with pretty straightforward methods.”
AGI arriving within a few years (two or three rather than literally next year) is a scenario worth taking seriously and planning for, even though Schulman doesn't think it will happen next year.
“First of all, I don't think this is going to happen next year but it's still useful to have the conversation. It could be two or three years instead.”
Humans likely use the same mental machinery to plan one month, one year, or a hundred years out—using language to describe all timescales rather than literal reinforcement learning with discount factors—so models may similarly develop capabilities that work across multiple horizons, possibly via phase transitions rather than clean scaling.
“We probably use the same mental machinery thinking about one month from now, one year from now, or a hundred years from now. We're not actually doing some kind of reinforcement learning where we need to worry about a discount factor that covers that timescale”
Chatbot verbosity and stylistic ticks likely arise partly from labeling biases (raters preferring verbose, structured answers and big info dumps), partly from training on one message at a time rather than full interactions (so a short clarifying-question reply looks less complete), and possibly partly from genuine user preference; some convergence may even come from unintentional distillation when human labelers feed tasks into chatbots.
“There might be some biases in the labeling that lead to verbosity. There's the fact that we tend to train for one message at a time rather than the full interaction. If you only see one message, then something that just has a clarifying question... is going to look less complete”
Base models have a probabilistic model of the whole world, so one could set up simulated questionnaires or conversations and examine how traits correlate, potentially replicating notable social-science results like moral foundations or the Asch conformity test by prompting base models in different ways.
“These models have a probabilistic model of the whole world and you can set up a simulated questionnaire or conversation and look at how anything is correlated... It'd be pretty cool to see if people could replicate some of the more notable results in social science, like moral foundations”
People who excel at RL research have a view of the whole stack—from RL algorithms to data collection and annotation processes—and curiosity about each part, balancing being empirical (letting experiments update views) with thinking from first principles about what the ideal data would be assuming learning works.
“the people who do well at this kind of research have some view of the whole stack and have a lot of curiosity about the different parts of it. You want to be both empirical and let experiments update your views, but you also want to think from first principles.”
Schulman's median timeline for AI replacing his own job is about five years.
“Oh, it replaces my job? Maybe five years.”
AI has progressed faster than Schulman expected since GPT-2; although he was bought into scaling and pre-training, GPT-2 didn't fully convince him, and it was only after GPT-3 that he pivoted his and his team's focus fully to language models.
“Faster than I expected since GPT-2. I was pretty bought into scaling and pre-training being a good idea. But when GPT-2 was done, I wasn't completely sold on it being revolutionizing everything. It was really after GPT-3 that I pivoted what I was working on”
Once vision capabilities improve, models will be able to use websites designed for humans without redesign, so the web won't be totally redesigned with APIs everywhere; though some sites will benefit from better text-based representations and clear indication of interactable elements for AI use.
“I expect that models will be able to use websites that are designed for humans just by using vision, after the vision capabilities get a bit better. So there wouldn't be an immediate need to change them.”