causal
Batching trades per-user latency for aggregate throughput
Batching many users into one GPU pass maximizes parallel utilization, but each batch carries more total work per step, so any single user's request spends longer on the GPU, a trade-off inherent to GPU design.
causalpending
Speaker
Neil MovvaEvidence Quote
“you're doing net more work when you run a large batch of compute together.”
Created: 11/17/2026, 6:43:49 AM
Your notes
Loading notes...