forecast
Separation of Training and Inference Compute
There will be a separation of training and inference compute as the field stabilizes; NVIDIA's Vera Rubin is designed for this with a new chip with very little high-bandwidth memory optimized for pre-fill (the matrix-multiplication-heavy part of inference), lowering cost of ownership per FLOP, while memory is only needed during autoregressive generation with KV cache swaps.
forecastpending
Speaker
Nathan LambertEvidence Quote
“they have this new GPU that’s designed for that specific use case, and then the cost of ownership per flop is actually way lower.”
Created: 6/13/2026, 3:24:41 AM
Your notes
Loading notes...