Ilya Sutskever – We're moving from the age of scaling to the age of research

AI should be built to robustly care for sentient life generally rather than specifically for humans, because: (1) AI systems will eventually be the majority of sentient beings; (2) if an AI cares about sentience, it would care about its own existence as well, creating intrinsic alignment; (3) mirror neurons suggest empathy emerges from modeling others with the same circuits used for self-modeling.

“It's the AI that's robustly aligned to care about sentient life specifically... I think in particular, there's a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient... even if you got an AI to care about sentient beings... it would still be the case that most sentient beings will be AIs. There will be trillions, eventually quadrillions, of AIs. Humans will be a very small fraction of sentient beings.”

Humans should be thought of as agents learning from experience rather than passive vessels, which suggests neural networks should also learn from data rather than relying solely on hard-coded knowledge.

“The brain learns from experience, therefore our neural net should learn from experience. The idea that the brain responds to experience therefore our neural net should learn from experience.”

Value functions—intermediate rewards that signal whether a system is on a promising path before reaching a final goal—are fundamental to efficient learning but are underutilized in current LLM training because long-horizon tasks make learning from final outcomes inefficient.

“The value function says something like, 'Maybe I could sometimes, not always, tell you if you are doing well or badly.' The notion of a value function is more useful in some domains than others. For example, when you play chess and you lose a piece, I messed up. You don't need to play the whole game to know that what I just did was bad... The value function lets you short-circuit the wait until the very end.”

The word 'scaling' is linguistically powerful and shaped the field's thinking by directing everyone to pursue scaling recipes, but now that scale is very large, 100x more compute would not solve fundamental problems like generalization, requiring different approaches.

“This is an example of how language affects thought. 'Scaling' is just one word, but it's such a powerful word because it informs people what to do. They say, 'Let's try to scale things.'... Is the belief really, 'Oh, it's so big, but if you had 100x more, everything would be so different?' It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don't think that's true.”

The term 'AGI' was coined as a reaction to 'narrow AI,' creating a false dichotomy and misdirecting AI development toward general-purpose systems, when the actual important frontier is continual learning—creating systems that can learn anything rather than systems that know everything.

“The reason that the term AGI exists is, in my opinion, not so much because it's a very important, essential descriptor of some end state of intelligence, but because it is a reaction to a different term that existed, and the term is narrow AI... So in response, as a reaction to this, some people said, this is not good. It is so narrow. What we need is general AI, an AI that can just do all the things... But if you think about the term 'AGI', especially in the context of pre-training, you will realize that a human being is not an AGI... Instead, we rely on continual learning.”

The term 'superintelligence' and concept of 'pre-training' have become semantic anchors that shape thinking, limiting consideration of alternatives; superintelligence as learning efficiency plus deployment is different from superintelligence as omniscience.

“The second thing that got a lot of traction is pre-training, specifically the recipe of pre-training... I think the way people do RL now is maybe undoing the conceptual imprint of pre-training... the thing that happened with AGI and pre-training is that in some sense they overshot the target.”

In a long-run equilibrium with powerful AIs, one concern is that humans become passive consumers of AI services rather than active participants, causing loss of agency; this could be addressed through neural interfaces (Neuralink++) that allow humans to directly share understanding with AIs.

“I'm going to preface by saying I don't like this solution, but it is a solution. The solution is if people become part-AI with some kind of Neuralink++. Because what will happen as a result is that now the AI understands something, and we understand it too, because now the understanding is transmitted wholesale. So now if the AI is in some situation, you are involved in that situation yourself fully.”

Self-play training as traditionally implemented is too narrow, only developing negotiation, conflict, and strategic skills, limiting its applicability to broader AI development.

“The thing is that self-play, at least the way it was done in the past—when you have agents which somehow compete with each other—it's only good for developing a certain set of skills. It is too narrow. It's only good for negotiation, conflict, certain social skills, strategizing, that kind of stuff.”

Reinforcement learning training inadvertently biases models toward benchmark performance by having researchers design RL environments that improve eval scores, causing reward hacking where models overfit to narrow evaluation criteria rather than developing robust generalization.

“One thing you could do, and I think this is something that is done inadvertently, is that people take inspiration from the evals. You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' I think that is something that happens, and it could explain a lot of what's going on.”

Superintelligence should be understood not as a finished omniscient mind but as an extremely efficient learner with human-level learning capability that would be deployed into the world like a highly motivated 15-year-old—capable but novice, requiring continual learning and development across different domains.

“I produce a superintelligent 15-year-old that's very eager to go. They don't know very much at all, a great student, very eager. You go and be a programmer, you go and be a doctor, go and learn. So you could imagine that the deployment itself will involve some kind of a learning trial-and-error period. It's a process, as opposed to you dropping the finished thing.”

The field of AI has transitioned from an era of research (2012-2020) where compute was the bottleneck, to an era of scaling (2020-2025) where scaling recipes dominated thinking, and is now entering a new era of research where compute is abundant but ideas are scarce.

“Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling... But now the scale is so big... So it's back to the age of research again, just with big computers.”

There exists a fundamental machine learning principle that would enable human-like learning but cannot be discussed freely in the current environment, limiting what can be said publicly about solutions to generalization problems.

“That is a great question to ask, and it's a question I have a lot of opinions about. But unfortunately, we live in a world where not all machine learning ideas are discussed freely, and this is one of them. There's probably a way to do it. I think it can be done... I think it points to the existence of some machine learning principle that I have opinions on. But unfortunately, circumstances make it hard to discuss in detail.”

Deployed continual learning AI instances across the economy can functionally become superhuman through shared experience aggregation without requiring recursive self-improvement, because a single model learning across all economic tasks simultaneously accumulates knowledge that no individual human can.

“if you have a single model—this is explicitly your vision—where instances of a model which are deployed through the economy doing different jobs, learning how to do those jobs, continually learning on the job, picking up all the skills that any human could pick up, but picking them all up at the same time, and then amalgamating their learnings, you basically have a model which functionally becomes superintelligent even without any sort of recursive self-improvement in software.”

Research taste in AI comes from aesthetic judgment about how AI should work based on how people work, using principles like: artificial neurons should mimic brain neurons, models should learn from experience like brains, representations should be distributed, and designs should reflect beauty and simplicity.

“One thing that guides me personally is an aesthetic of how AI should be, by thinking about how people are, but thinking correctly... The idea of the artificial neuron is directly inspired by the brain... The idea of the distributed representation... The brain learns from experience, the neural net should learn from experience... looking for almost beauty, beauty and simplicity. Ugliness, there's no room for ugliness. It's beauty, simplicity, elegance, correct inspiration from the brain.”

Capping the power of the most powerful superintelligence through some form of agreement or constraint would materially help address alignment concerns, though the mechanism for doing this is unclear.

“I think it would be really materially helpful if the power of the most powerful superintelligence was somehow capped because it would address a lot of these concerns. The question of how to do it, I'm not sure, but I think that would be materially helpful when you're talking about really, really powerful systems.”

Evolution has provided humans with strong priors for motor skills and perception (vision, hearing, locomotion) through millions of years of selection, but has not provided comparable priors for language, math, and coding, suggesting that human learning superiority in those domains reflects better learning algorithms rather than better domain-specific instincts.

“For things like vision, hearing, and locomotion, I think there's a pretty strong case that evolution has given us a lot... But in language and math and coding, probably not. It still seems better than models... But are they better than the average human at learning? Oh yeah. Oh yeah, absolutely... language, math, and coding—and especially math and coding—suggests that whatever it is that makes people good at learning is probably not so much a complicated prior, but something more, some fundamental thing.”

The main advantage of SSI's straight-shot superintelligence approach is insulation from market competition and the ability to focus purely on research without trade-offs, though there is a counterpoint that gradual deployment of intermediate AI systems helps the public and world understand and prepare for powerful AI.

“The case for is that one of the challenges that people face when they're in the market is that they have to participate in the rat race... It is nice to say, 'We'll insulate ourselves from all this and just focus on the research and come out only when we are ready, and not before.' But the counterpoint is valid too, and those are opposing forces. The counterpoint is, 'Hey, it is useful for the world to see powerful AI.'”

Gradual deployment of AI systems is analogous to how airlines improved safety through deployment experience rather than theory alone—airplanes are safer today through real-world deployment, discovery of failures, and correction rather than through pure pre-deployment safety thinking.

“I can't think of another discipline in human engineering and research where the end artifact was made safer mostly through just thinking about how to make it safe, as opposed to, why airplane crashes per mile are so much lower today than they were decades ago. Why is it so much harder to find a bug in Linux than it would have been decades ago? I think it's mostly because these systems were deployed to the world. You noticed failures, those failures were corrected and the systems became more robust.”

Evolution has encoded sophisticated high-level social desires (caring about social status, being liked, being in good standing) into human brains, but it is mysterious how evolution could encode these abstract high-level concepts when it primarily has access to low-level signals like chemical smell.

“evolution also has endowed us with all these social desires. We really care about being seen positively by society. We care about being in good standing. All these social intuitions that we have, I feel strongly that they're baked in. I don't know how evolution did it because it's a high-level concept that's represented in the brain... It's not a low-level signal like smell... The brain needs to do a lot of processing to piece together lots of bits of information to understand what's going on socially. Somehow evolution said, 'That's what you should care about.' How did it do it?”

Humans are semi-RL agents who pursue rewards until emotional satiation causes them to shift to different rewards, the market is short-sighted, evolution is intelligent but limited, and governments are designed as never-ending conflicts—these structural limitations make pure RL agents potentially dangerous.

“I think human beings are semi-RL agents. We pursue a reward, and then the emotions or whatever make us tire out of the reward and we pursue a different reward. The market is a very short-sighted kind of agent... The government has been designed to be a never-ending fight between three parts, which has an effect.”

Superintelligence alignment is fundamentally difficult because value learning is fragile, value optimization is fragile, and generalization is fragile—all three are instances of the same underlying generalization problem.

“One of the things that you could say about what causes alignment to be difficult is that your ability to learn human values is fragile. Then your ability to optimize them is fragile. You actually learn to optimize them. And can't you say, 'Are these not all instances of unreliable generalization?'”

SSI raised $3 billion in funding, but most compute at larger companies like OpenAI and Anthropic goes to inference and product engineering rather than research, so when adjusted for product costs, SSI's research compute is comparable to frontier labs.

“SSI has raised $3 billion, which is a lot by any absolute sense. But you could say, 'Look at the other companies raising much more.' But a lot of their compute goes for inference. These big numbers, these big loans, it's earmarked for inference. That's number one. Number two, if you want to have a product on which you do inference, you need to have a big staff of engineers, salespeople. A lot of the research needs to be dedicated to producing all kinds of product-related features.”

Teenagers learn to drive in 10 hours with minimal explicit rewards or verification, instead relying on unsupervised experience, internalized sense of progress, and robust value functions; achieving this in AI would require fundamentally different training approaches than current supervised or RL paradigms.

“A teenager learning how to drive a car is not exactly getting some prebuilt, verifiable reward. It comes from their interaction with the machine and with the environment. It takes much fewer samples. It seems more unsupervised. It seems more robust?... Much more robust. The robustness of people is really staggering.”

Pre-training works because it discovered a powerful recipe (mix compute + data + neural net size) that scales reliably, reducing investment risk by making progress predictable, whereas research is fundamentally uncertain.

“This is also great. Companies love this because it gives you a very low-risk way of investing your resources. It's much harder to invest your resources in research. Compare that. If you research, you need to be like, 'Go forth researchers and research and come up with something', versus get more data, get more compute. You know you'll get something from pre-training.”

Showing AI systems to the public is essential for helping people understand their power, whereas abstract essays about AI capabilities fail to communicate the reality because people cannot imagine systems so different from daily experience.

“if it's hard to imagine, what do you do? You've got to be showing the thing. You've got to be showing the thing. I maintain that most people who work on AI also can't imagine it because it's too different from what people see on a day-to-day basis.”

First-mover advantages for human-level learning AI may be temporary and partial, with imitator companies eventually replicating the capability despite first-mover market advantages.

“One company produced an advance and the other company scrambled and produced some similar things after some amount of time and they started to compete in the market and push the prices down.”

Models trained intensively on competitive programming problems become highly specialized at that narrow domain and fail to generalize to other coding tasks, unlike humans who practice 100 hours of competitive programming and retain broader coding skills.

“One of them decided they want to be the best competitive programmer, so they will practice 10,000 hours for that domain... By doing so, they became one of the best. Student number two thought, 'Oh, competitive programming is cool.' Maybe they practiced for 100 hours, much less, and they also did really well. Which one do you think is going to do better in their career later on? The second... The models are much more like the first student, but even more. Because then we say, the model should be good at competitive programming so let's get every single competitive programming problem ever.”

Self-play is most useful for narrow domains like competition and social negotiation, but adversarial setups more broadly (debate, prover-verifier, LLM-as-judge) can create diversity by incentivizing agents to differentiate their approaches from each other.

“The thing is that self-play, at least the way it was done in the past... it's only good for developing a certain set of skills. It is too narrow. It's only good for negotiation, conflict, certain social skills, strategizing... Actually, I think that self-play did find a home, but just in a different form. So things like debate, prover-verifier, you have some kind of an LLM-as-a-Judge which is also incentivized to find mistakes in your work.”

Emotions function as robust value functions that were shaped by evolution over millions of years, providing humans with reliable intermediate signals about whether they are making good decisions without requiring explicit external feedback.

“Whatever the human value function is, with a few exceptions around addiction, it's actually very, very robust. So for something like a teenager that's learning to drive, they start to drive, and they already have a sense of how they're driving immediately, how badly they are, how unconfident.”

Frontier AI companies will eventually converge on alignment strategies even if they diverge on technical approaches, and this convergence will happen as AI becomes more visibly powerful and companies recognize the need for shared safety standards.

“I maintain that in the end there will be a convergence of strategies. I think there will be a convergence of strategies where at some point, as AI becomes more powerful, it's going to become more or less clearer to everyone what the strategy should be... I think that this time, if not already, all the other companies will realize that they're striving towards the same thing.”

Copies of a single human researcher will show diminishing returns because diversity of thinking is more valuable than parallelization, suggesting that a million copies of the same AI will produce less value than diverse teams of different AIs.

“If there were literal copies of me, I'm not sure how much more incremental value you'd get. People who think differently, that's what you want.”

LLM models from different companies trained on different datasets are remarkably similar to each other, suggesting that either datasets are more overlapping than they appear or that the pre-training optimization process converges on similar solutions regardless of training data.

“Why is it that if you look at different models, even released by totally different companies trained on potentially non-overlapping datasets, it's actually crazy how similar LLMs are to each other? Maybe the datasets are not as non-overlapping as it seems.”

What makes an idea worth pursuing is a combination of properties: beauty, simplicity, elegance, and correct inspiration from the brain—these properties need to be present simultaneously, and their presence gives confidence in a top-down belief that can sustain you through contradicting data.

“Beauty, simplicity, elegance, correct inspiration from the brain. All of those things need to be present at the same time. The more they are present, the more confident you can be in a top-down belief. The top-down belief is the thing that sustains you when the experiments contradict you.”

Pre-training does not require thoughtful selection of training data because it uses all available data, whereas RL training requires explicit choices about which environments to create, introducing many degrees of freedom and opportunities for misalignment with actual capabilities.

“When you do pre-training, the question of what data to train on was answered, because that answer was everything. When you do pre-training, you need all the data. So you don't have to think if it's going to be this data or that data. But when people do RL training, they do need to think. They say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' From what I hear, all the companies have teams that just produce new RL environments and just add it to the training mix. The question is, well, what are those? There are so many degrees of freedom.”

Even with current approaches stalling out, companies could still generate tremendous revenue and make profits, suggesting that revenue-generating stall-out is distinct from technical stall-out.

“I think stalling out will look like…it will all look very similar among all the different companies. It could be something like this. I'm not sure because I think even with stalling out, I think these companies could make a stupendous revenue. Maybe not profits because they will need to work hard to differentiate each other from themselves, but revenue definitely.”

Rapid economic growth is very possible with broad deployment of efficient learning AI, but actual growth rate depends on implementation constraints and real-world friction; different countries with different regulatory rules will have different deployment speeds, making growth unpredictable.

“I think with broad deployment, there are two arguments you could make which are conflicting. One is that once indeed you get to a point where you have an AI that can learn to do things quickly and you have many of them, then there will be a strong force to deploy them in the economy unless there will be some kind of a regulation that stops it... I think very rapid economic growth is possible. We will see all kinds of things like different countries with different rules and the ones which have the friendlier rules, the economic growth will be faster. Hard to predict.”

The possibility that human neurons perform substantially more computation than current models suggest would create an additional blocker to replicating human learning in silicon-based systems.

“There may be another blocker though, which is that there is a possibility that the human neurons do more compute than we think. If that is true, and if that plays an important role, then things might be more difficult.”

Current AI training regimes produce models with neither the sample efficiency nor the robustness of humans because they lack both the evolved inductive biases and the sophisticated value functions that guide human learning.

“Somehow a human being, after even 15 years with a tiny fraction of the pre-training data, they know much less. But whatever they do know, they know much more deeply somehow. Already at that age, you would not make mistakes that our AIs make.”

Self-play is valuable primarily because it creates data using only compute without requiring additional internet text data, making it attractive if data scarcity becomes the bottleneck.

“The reason why I thought self-play was interesting is because it offered a way to create models using compute only, without data. If you think that data is the ultimate bottleneck, then using compute only is very interesting.”

SSI's technical approach to AI differs from competitors in specific ways that Ilya believes are promising, but the exact approaches cannot be fully disclosed due to competitive concerns.

“What distinguishes SSI is its technical approach. We have a different technical approach that I think is worthy and we are pursuing it.”