forecast

Separation of Training and Inference Compute

There will be a separation of training and inference compute as the field stabilizes; NVIDIA's Vera Rubin is designed for this with a new chip with very little high-bandwidth memory optimized for pre-fill (the matrix-multiplication-heavy part of inference), lowering cost of ownership per FLOP, while memory is only needed during autoregressive generation with KV cache swaps.

forecastpending

Speaker

Nathan Lambert

Evidence Quote

“they have this new GPU that’s designed for that specific use case, and then the cost of ownership per flop is actually way lower.”

Source

Ai Sota #2026— Lex Fridman Podcast
Created: 6/13/2026, 3:24:41 AM

Your notes

Loading notes...