causal

Batching trades per-user latency for aggregate throughput

Batching many users into one GPU pass maximizes parallel utilization, but each batch carries more total work per step, so any single user's request spends longer on the GPU, a trade-off inherent to GPU design.

causalpending

Speaker

Neil Movva

Evidence Quote

“you're doing net more work when you run a large batch of compute together.”

Source

Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]— Invest Like the Best
Created: 11/17/2026, 6:43:49 AM

Your notes

Loading notes...