inference compute
Computation spent at answer time (running the model), as opposed to training-time compute.
As defined in Aravind Srinivas
Where it comes up · 4
- Artificial General Intelligence's Five Hard National Security Problems2 claims
- Aravind Srinivas2 claims
- Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job1 claim
- Cursor Team1 claim
Most important claims · 6 of 6Search all claims for “inference compute” →
- The real concern with powerful AGI is not models going rogue or the danger of open weights, but the concentration of power in the few individuals, companies, or nations who can afford the enormous inference compute to run the most valuable research jobs — so regulation should focus on application and access to compute, not on weights being dangerous.Aravind Srinivas
- If we can build an AI that, like Einstein or Feynman, can say 'I don't know,' go do a week of research, and come back with a dramatically better answer as more inference compute is applied, that scaling of inference compute will mark the beginning of real reasoning breakthroughs.Aravind Srinivas
- Software like large language models, despite their size, can be readily sent over the internet, making proliferation easier than hardware; the trickier constraint is computing power for inference, which scales in thousands to hundreds of thousands of dollars rather than millions or billions, meaning policy decisions about inference compute proliferation have significant impact.Artificial General Intelligence's Five Hard National Security Problems
- Around 30 to 50 percent of a lab's compute goes to inference, and that compute currently does nothing to improve the model, which Patel calls a waste.Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job
- Policy decisions about where inference compute is deployed and under what conditions could be supported by technical solutions such as hardware-enabled mechanisms where chips contain location attestation so that if chips are deployed where they shouldn't be, authorities can detect this.Artificial General Intelligence's Five Hard National Security Problems
- Scaling curves have more dimensions than the original compute, parameters, and data — notably inference compute and context length — and for long context with cheap inference, training an SSM may be worth a 10x worse training-scaling property because the inference budget for long context windows matters most.Cursor Team