Inference vs training compute
The split of a lab's computing resources between running trained models for users (inference) and building new models (training).
As defined in Why smarter AI models could drive up compute prices 10x
Where it comes up · 1
Most important claims · 3 of 3Search all claims for “Inference vs training compute” →
- Labs prefer not to shift compute toward inference because heavy inference spending signals that AI progress has stalled and that the lab is becoming a cloud provider, which is a less compelling business than building AGI, so labs want to keep most compute on training the next model.Why smarter AI models could drive up compute prices 10x
- Lab compute is growing only about 3x year over year, so a lab that keeps growing revenue 10x per year while compute grows 3x must resolve the gap through one of three channels: higher lab margins, higher compute prices, or a larger share of compute devoted to inference rather than training.Why smarter AI models could drive up compute prices 10x
- Anthropic's inference gross margins reportedly rose from about 40 percent in the middle of last year to upwards of 80 percent now for its model, and this margin expansion is one of the ways the revenue-compute gap is already being closed.Why smarter AI models could drive up compute prices 10x