causal

Inference Demand Is Constrained by Concurrency Not Just Speed

Serving giant models to billions of users requires many chips to communicate quickly with each other, because fast intra-cluster communication alone is insufficient at tens of thousands of chips, and infrastructure must be built across the full stack from wafer to watt.

causalpending

Speaker

Rob

Evidence Quote

“they're not going to be able to serve giant models at 10,000, 20,000 tokens per second.”

Source

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]— Invest Like the Best
Created: 11/17/2026, 7:08:01 AM

Your notes

Loading notes...