causal
Batching Amortizes Weight-Loading Cost
Inference is memory-bound because each parameter must be loaded from memory and used once, so batching many users to reuse each loaded weight amortizes the expensive load, and a transformer ASIC can batch around 2,500 users versus about 64 on GPUs.
causalpending
Speaker
GavinEvidence Quote
“You can have a much, much larger batch, not 64 but 2,500.”
Source
Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]— Invest Like the BestCreated: 11/17/2026, 12:35:53 PM
Your notes
Loading notes...