tokens per second
A measure of how fast a language model generates text, indicating the number of individual tokens (words or subwords) produced by the model in one second
As defined in Feed Your OWN Documents to a Local Large Language Model!
Other definitions
Speed at which a model generates output tokens; a usability measure for responsive AI.
Nvidia v. Google, Scaling Laws, and the Economics of AI
Where it comes up · 6
- Feed Your OWN Documents to a Local Large Language Model!2 claims
- Capital, Compute & AI Scaling1 claim
- Nvidia v. Google, Scaling Laws, and the Economics of AI1 claim
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]1 claim
- Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]1 claim
- Inference, Diffusion, World Models, and More | YC Paper Club1 claim
Most important claims · 7 of 7Search all claims for “tokens per second” →
- Inference in one to three years will be seen as a capability lever rather than a cost or convenience lever, because if an algorithm's performance scales with thinking time, then tokens per second directly equals peak intelligence delivered.Inference, Diffusion, World Models, and More | YC Paper Club
- Serving giant models to billions of users requires many chips to communicate quickly with each other, because fast intra-cluster communication alone is insufficient at tens of thousands of chips, and infrastructure must be built across the full stack from wafer to watt.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- A dual Nvidia RTX 6000 system can generate over 300 tokens per second when running a 1 billion parameter model like Llama 3.2, sustaining approximately 324 tokens per second on longer text generation tasks.Feed Your OWN Documents to a Local Large Language Model!
- A dual Nvidia RTX 6000 workstation can generate 324 tokens per second with a 1 billion parameter model, sustaining over 300 tokens per second consistently.Feed Your OWN Documents to a Local Large Language Model!
- In about three years, phones will run pruned versions of frontier models at 30 to 60 tokens per second for free, with privacy and no cloud cost, and if that is good enough, cloud demand for compute is at risk; this is the most plausible and scariest bear case.Nvidia v. Google, Scaling Laws, and the Economics of AI
- To enable the next wave of uses such as tree search and speech conversation, inference must become roughly 100x cheaper, latency must drop by an order of magnitude, and systems should generate about 1,000 tokens per second so code writing takes seconds instead of tens of seconds.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Cerebras reports Llama 3.1 405B inference above 900 tokens per second, roughly 70 to 75 times faster than GPUs for inference, showing order-of-magnitude gains for inference-focused hardware.Capital, Compute & AI Scaling