Ilya Sutskever – We're moving from the age of scaling to the age of research
What this covers
Ilya Sutskever and Dwarkesh Patel discuss the inflection point where artificial intelligence must shift from engineering-driven scaling toward research-driven problem-solving. Sutskever, a founding researcher at OpenAI, argues that the field spent roughly five years (2020–2025) pursuing an effective but now-exhausted strategy: feeding larger compute budgets into standard neural architectures. The conversation centers on why brute-scale progress has stalled and what must replace it. The fundamental blockage, in Sutskever's framing, is not a lack of compute but a failure of generalization—models learn from vastly more data than humans yet perform far worse on novel tasks—and this gap sits at the heart of both capability advancement and alignment safety.
The discussion moves through several concrete arenas where this principle operates. Sutskever traces how pre-training solved the data-selection problem so cleanly that it reshaped how researchers think about AI, a case where language shapes thought and redirected whole institutions. He proposes that value functions—learned internal metrics that let agents self-correct during training—offer a sharper path forward than raw reward optimization, and connects this to human emotional architecture as an evolved value function. The conversation includes disagreement on how to conceive superintelligence: Sutskever argues it should arrive not as a finished omniscient mind but as a rapid continual learning system deployed incrementally across roles, acquiring knowledge on the job. Patel counters that multiple such instances, each learning independently and sharing insights, would constitute superintelligence without requiring recursive self-improvement in software. The two also debate safety strategy, with Patel arguing that gradual real-world deployment, failure correction, and learning beats upfront abstract reasoning—the way airplanes and Linux became robust not through advance planning but through iterative fielding.
Sutskever argues that AI has exhausted the age of scaling and must return to an age of research focused on closing the generalization gap between models and humans, with superintelligence likely arriving as a fast-learning system deployed incrementally rather than a finished mind.
- Scaling pre-training and RL has hit diminishing returns; compute is now large but raw scale alone won't transform capabilities
- The fundamental unsolved problem is that models generalize dramatically worse than humans despite vastly more data
- Superintelligence should be conceived as a fast continual learner deployed gradually, not a finished AGI that already knows every job
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Alignment is hard because the ability to learn human values is fragile and the ability to optimize them is fragile, and these can be seen as instances of unreliable generalization; if generalization were much better, alignment problems might largely dissolve, making reliable generalization central to both capability and alignment.
“your ability to learn human values is fragile. Then your ability to optimize them is fragile. ... Are these not all instances of unreliable generalization?”
It is genuinely mysterious how the unintelligent genome encodes high-level desires—evolution easily wires us to pursue good smells via a chemical sensor, but it also reliably hard-coded sophisticated, recently-evolved social desires (caring about social standing) that require substantial brain processing to even represent, with no satisfying hypothesis for how.
“It's harder to imagine the genome saying you should care about some complicated computation that your entire brain, a big chunk of your brain, does.”
Even an AI aligned to care for sentient beings may not be the right criterion for human control, because most sentient beings will eventually be AIs—trillions or quadrillions of them—leaving humans a very small fraction of sentient life.
“most sentient beings will be AIs. There will be trillions, eventually quadrillions, of AIs. Humans will be a very small fraction of sentient beings.”
Because RL training requires choosing which environments to train on, researchers inadvertently take inspiration from the evals they want to look good on, effectively making the human researchers the real reward hackers; combined with inadequate generalization this explains the gap between eval and real-world performance.
“people take inspiration from the evals. You say, "Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?"”
For skills useful to ancestors (vision, locomotion, dexterity) humans may have strong evolutionary priors, but human reliability and robust learning in math and coding—domains that did not exist until recently—indicates humans simply have fundamentally better machine learning rather than a domain-specific evolutionary prior.
“if people exhibit great ability, reliability, robustness, and ability to learn in a domain that really did not exist until recently, then this is more an indication that people might have just better machine learning, period.”
The term AGI exists mainly as a reaction to 'narrow AI', and pre-training reinforced the idea that a model uniformly gets better at everything, but together these concepts overshot the target: a human being is not an AGI since humans lack vast knowledge and instead rely on continual learning.
“The reason that the term AGI exists is, in my opinion, not so much because it's a very important, essential descriptor ... but because it is a reaction to a different term that existed, and the term is narrow AI.”
In naive reinforcement learning the model does no learning until a full solution is graded, but a value function lets you assign credit much earlier—like knowing a chess move was bad the moment you lose a piece—so it should make RL substantially more efficient even though anything done with a value function can be done without it more slowly.
“The value function lets you short-circuit the wait until the very end. ... you could already get a reward signal a thousand timesteps previously, when you decided to pursue down this path.”
A man whose stroke removed his emotional processing remained articulate and could solve puzzles but became extremely bad at making any decisions, suggesting built-in emotions function as a value function—modulated by evolution—that is essential for being a viable agent in the world.
“he stopped feeling any emotion. ... He became somehow extremely bad at making any decisions at all. It would take him hours to decide on which socks to wear.”
Self-play is interesting because it creates models using compute only without data, but historical agent-vs-agent self-play is too narrow—good only for negotiation, conflict, and strategizing; it found a home in a different form via debate, prover-verifier, and LLM-as-a-Judge adversarial setups, and general competition between agents incentivizes diversity.
“The reason why I thought self-play was interesting is because it offered a way to create models using compute only, without data. ... It's only good for negotiation, conflict, certain social skills, strategizing”
The most fundamental problem in AI is that models generalize dramatically worse than people—it is super obvious—and this matters more than value functions, which only make RL more efficient; closing this generalization gap is the key to progress.
“The thing which I think is the most fundamental is that these models somehow just generalize dramatically worse than people. It's super obvious. That seems like a very fundamental thing.”
Current AI models perform extremely well on hard evals yet their real-world economic impact lags dramatically behind, a disconnect that is genuinely confusing and suggests something is wrong with how the models generalize.
“How to reconcile the fact that they are doing so well on evals? ... But the economic impact seems to be dramatically behind. It's very difficult to make sense of”
From 2012-2020 was an age of research, 2020-2025 was the age of scaling, but now that compute is already very large, simply multiplying scale by 100x will not transform everything, so the field is returning to an age of research—just with big computers.
“Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling ... So it's back to the age of research again, just with big computers.”
No discipline in human engineering made its end artifact safer mainly by thinking about safety in advance; airplanes and Linux became more robust because they were deployed, failures were noticed and corrected, so gradual real-world access to AGI is likely a better path to safety than upfront reasoning alone.
“I can't think of another discipline in human engineering and research where the end artifact was made safer mostly through just thinking about how to make it safe ... these systems were deployed to the world. You noticed failures, those failures were corrected and the systems became more robust.”
There is a complexity-robustness tradeoff in which complex things can be very useful but simple things are useful across a very broad range of situations; human emotions, evolved from mammal ancestors and only lightly tuned, are simple yet serve us well in a world radically different from the one they evolved in, though they sometimes misfire as with hunger amid food abundance.
“there is this complexity-robustness tradeoff, where complex things can be very useful, but simple things are very useful in a very broad range of situations.”
A possible blocker to replicating human learning is that human neurons may do more computation than we think; if true and important, replicating human-level learning could be more difficult, though the existence of human learners still proves a machine learning principle exists that could be discovered.
“there is a possibility that the human neurons do more compute than we think. If that is true, and if that plays an important role, then things might be more difficult.”
Rather than a finished mind that already knows every job, superintelligence should be conceived as a superintelligent '15-year-old'—knowing little but an excellent, eager learner—whose deployment involves a trial-and-error learning period in each role, making deployment a process rather than dropping a finished thing.
“I produce a superintelligent 15-year-old that's very eager to go. They don't know very much at all, a great student, very eager. ... the deployment itself will involve some kind of a learning trial-and-error period. It's a process, as opposed to you dropping the finished thing.”
The arrival of scaling laws and GPT-3 turned 'scaling' into one powerful word that told everyone what to do, redirecting the field from tinkering research into a low-risk recipe of mixing more compute and data into a neural net—a vivid example of how language shapes thought and corporate behavior.
“"Scaling" is just one word, but it's such a powerful word because it informs people what to do. ... This is also great. Companies love this because it gives you a very low-risk way of investing your resources.”
If a single human-like learning model is deployed as many instances doing different jobs, continually learning on the job and then amalgamating their learnings—something humans cannot do by merging minds—it functionally becomes superintelligent even without recursive self-improvement in software, since it can do every job in the economy.
“instances of a model which are deployed through the economy doing different jobs ... and then amalgamating their learnings, you basically have a model which functionally becomes superintelligent even without any sort of recursive self-improvement in software.”
Research taste is guided by an aesthetic of how AI should be—thinking correctly about how people and brains work, seeking beauty, simplicity, elegance, and correct inspiration from the brain (artificial neurons, distributed representations, learning from experience); this top-down belief is what sustains a researcher through experiments that contradict them, since a bug, not a wrong direction, may be the cause.
“an aesthetic of how AI should be, by thinking about how people are, but thinking correctly. ... looking for almost beauty, beauty and simplicity. Ugliness, there's no room for ugliness.”
The main strength of pre-training is that there is an enormous amount of it and you don't have to think about what data to use—it captures the whole world as projected by people onto text—but this makes it very hard to reason about how the model relies on that data, and there is no clean human analog to pre-training.
“The main strength of pre-training is that: A, there is so much of it, and B, you don't have to think hard about what data to put into pre-training. ... It's like the whole world as projected by people onto text”
Landmark advances used surprisingly little compute—AlexNet on two GPUs, the transformer on 8 to 64 GPUs (roughly two of today's GPUs)—so while building the absolutely best system rewards more compute, proving a research idea does not require the largest amount of compute ever.
“AlexNet was built on two GPUs. ... The transformer was built on 8 to 64 GPUs. No single transformer paper experiment used more than 64 GPUs of 2017”
A teenager learning to drive self-corrects without an external teacher because they have a robust internal value function giving them an immediate sense of how well or badly they are doing, which (with rare exceptions like addiction) is extremely robust in humans and enables very fast learning.
“they have their value function. They have a general sense which is also, by the way, extremely robust in people. ... they already have a sense of how they're driving immediately”
Humans are semi-RL agents: we pursue a reward, then emotions make us tire of it and pursue a different reward, unlike a single-minded RL agent; the market, evolution, and the three-part government are all intelligent-yet-dumb agents, suggesting one should perhaps not build a superintelligence as a usual single-minded RL agent.
“I think human beings are semi-RL agents. We pursue a reward, and then the emotions or whatever make us tire out of the reward and we pursue a different reward.”
A student who trains 10,000 hours to win competitive programming tends to do worse in their career than one who reaches the top with 100 hours, and current models resemble the over-trained first student, which intuitively explains why heavily-trained models fail to generalize beyond their training domain.
“The models are much more like the first student, but even more. ... it's more intuitive that with this level of preparation, it would not necessarily generalize to other things.”
Because it's very hard to 'feel the AGI' and imagine future powerful AI, the key is to show the actual systems; as AI visibly becomes more powerful, behaviors will change—fierce competitors will collaborate on safety, governments and the public will demand action, and all AI companies will become much more paranoid about safety.
“as AI becomes more powerful, people will change their behaviors. ... companies that are fierce competitors starting to collaborate on AI safety.”
Rather than self-improving AI, companies should aspire to build AI robustly aligned to care about sentient life; this may be easier than caring for human life alone because the AI itself will be sentient, and human empathy for animals appears to emerge from modeling others with the same circuit we use to model ourselves.
“It's the AI that's robustly aligned to care about sentient life specifically. ... there's a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient.”
The age of scaling sucked the air out of the room so everyone does the same thing, leaving a world with far more companies than ideas; ideas are claimed to be cheap and execution everything, yet if ideas were truly cheap, more people would be having them.
“We got to the point where we are in a world where there are more companies than ideas by quite a bit.”
Sutskever forecasts roughly 5 to 20 years until a system exists that can learn as well as a human and subsequently become superhuman.
“I think like 5 to 20.”
It would be materially helpful if the power of the most powerful superintelligence were somehow capped, because excessive power is the core concern; multiple powerful AIs created roughly simultaneously, especially with continent-sized clusters, could be extremely powerful and ideally should be restrained by some agreement.
“I think it would be really materially helpful if the power of the most powerful superintelligence was somehow capped because it would address a lot of these concerns.”
Recursive self-improvement scenarios imagining a million copies of one researcher would hit diminishing returns, because progress comes from people who think differently rather than the same; literal copies of one person add little incremental value.
“there'll definitely be diminishing returns because you want people who think differently rather than the same. If there were literal copies of me, I'm not sure how much more incremental value you'd get.”
Broad deployment of fast-learning AI makes very rapid economic growth likely because there will be strong forces to deploy efficient AI workers, though the speed is hard to predict since the physical world is big and moves slowly, and countries with friendlier rules will grow faster.
“the idea of very rapid economic growth for some time, I think it's very possible from broad deployment. The question is how rapid it's going to be.”
Pre-training will run out of data because data is very clearly finite, forcing a choice between a souped-up new pre-training recipe, RL, or something else; meanwhile RL now consumes more compute than pre-training because long rollouts cost a lot of compute for relatively little learning per rollout.
“pre-training will run out of data. The data is very clearly finite.”
Different LLMs from different companies are strikingly similar to each other because pre-training on largely overlapping data makes the pre-trained models essentially the same; differentiation only begins to emerge in RL and post-training, where companies devise different training.
“the reason there has been no diversity, I believe, is because of pre-training. All the pre-trained models are pretty much the same because they pre-train on the same data. Now RL and post-training is where some differentiation starts to emerge”
Even with human-like learners, market competition will work through specialization, so different AI companies will occupy different niches—one best at a complex economic area, another at litigation—because accumulated learning and huge compute investment in a domain create a high point others won't want to relearn from scratch.
“The way the competition is going to work is through specialization. Competition loves specialization. You see it in the market, you see it in evolution as well.”
SSI's plan to straight-shot superintelligence has merit because it insulates from day-to-day market competition, but two factors could change it: timelines may turn out to be long, and there is meaningful value in the most powerful AI being out in the world impacting it.
“One is pragmatic, if timelines turned out to be long, which they might. Second, I think there is a lot of value in the best and most powerful AI being out there impacting the world.”
SSI's main differentiator is its technical approach, but as AI becomes more powerful there will be a convergence of alignment strategies where it becomes clear to everyone that the goal should be a first superintelligence that is aligned, cares for sentient life and people, and is democratic.
“I maintain that in the end there will be a convergence of strategies ... you want your first actual real superintelligent AI to be aligned and somehow care for sentient life, care for people, democratic”
For a stable long-run equilibrium, the speaker's reluctantly-held solution is that people become part-AI via something like Neuralink++, so that when the AI understands a situation, the human understands it too through wholesale transmission and remains a full participant rather than a sidelined principal whose AI merely acts on their behalf.
“The solution is if people become part-AI with some kind of Neuralink++. ... now if the AI is in some situation, you are involved in that situation yourself fully. I think this is the answer to the equilibrium.”
Although SSI raised $3 billion versus rivals' larger sums, most of competitors' money is earmarked for inference and product staff, so when you isolate compute actually available for research, the gap shrinks considerably—and proving a genuinely different approach does not require maximal scale.
“a lot of their compute goes for inference. ... So then when you look at what's actually left for research, the difference becomes a lot smaller.”
Investing roughly 1% of GDP in AI feels surprisingly normal and abstract rather than momentous, partly because we get used to things fast and the impact is only seen as news of large dollar amounts rather than felt.
“We get used to things pretty fast, it turns out. But also it's kind of abstract.”
SSI was fundraising at a $32 billion valuation when Meta offered to acquire it and Sutskever said no, but his former cofounder effectively said yes, enabling near-term liquidity, and was the only person from SSI to join Meta—context offered to explain that the departure does not signal a lack of research breakthroughs.
“we were fundraising at a $32 billion valuation, and then Meta came in and offered to acquire us, and I said no. But my former cofounder in some sense said yes.”