causal
Inference Demand Is Constrained by Concurrency Not Just Speed
Serving giant models to billions of users requires many chips to communicate quickly with each other, because fast intra-cluster communication alone is insufficient at tens of thousands of chips, and infrastructure must be built across the full stack from wafer to watt.
causalpending
Speaker
RobEvidence Quote
“they're not going to be able to serve giant models at 10,000, 20,000 tokens per second.”
Source
Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]— Invest Like the BestCreated: 11/17/2026, 7:08:01 AM
Your notes
Loading notes...