Flops
Floating-point operations per second, a measure of computational throughput.
As defined in dwarkesh_china_is_killing_the_us_on_energy_does_that_mean_t
Other definitions
Floating-Point Operations Per Second—a measure of computational throughput, here used to estimate the brain's compute (~10^15).
The Past and Future of AI (with Dwarkesh Patel)Floating-point operations, a measure of computational work used to compare model training compute to brain computation.
Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthroughFloating-point operations per second; the raw compute throughput of a chip.
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
Where it comes up · 16
- @Asianometry & Dylan Patel — How the semiconductor industry actually works5 claims
- Dwarkesh Patel Podcast - Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 20284 claims
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served3 claims
- Satya Nadella — How Microsoft is preparing for AGI3 claims
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]3 claims
- Deepseek Dylan Patel Nathan Lambert2 claims
- Ex-Google CEO: What Artificial Superintelligence Will Actually Look Like w/ Eric Schmidt & Dave B2 claims
- Andrej Karpathy — AGI is still a decade away1 claim
and 8 more
Most important claims · 10 of 31Search all claims for “Flops” →
- Model FLOPS utilization (MFU), the share of theoretical compute actually applied to training, runs around 35 to 40 percent, with published GPT-3, LaMDA and Megatron figures in the high 20s to high 30s, and Baker argues it is the best single metric for evaluating a lab.Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]
- Model FLOPS utilization (MFU) measures how much of the advertised peak FLOPS a chip actually delivers on real workloads, and GPUs often achieve only 20 to 50 percent depending on the workload, so the relevant metric is achieved rather than headline FLOPS.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- Setting weight-fetch time equal to weight-multiply time yields a hardware constant (FLOPs/byte ratio) of roughly 300 across recent GPUs, so the optimal batch size is approximately 300 times the model's sparsity ratio—e.g., DeepSeek activating 32 of 256 experts gives a sparsity of 8, implying a batch around 2,000-3,000 tokens; this depends only on sparsity, not model scale.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Because chips deployed this year, such as GB300, TPUv7 and Trainium3, deliver 3-5x more performance per watt than prior generations, each new gigawatt adds far more effective compute than older ones, so labs' share of usable FLOPs compounds beyond their share of watts.Dwarkesh Patel Podcast - Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 2028
- Labs are being practical with fixed flops and cost budgets: pre-training is not where you want most of your flops, so models have gotten smaller in the pre-training stage but make it up in reinforcement learning and mid-training to get the most bang for the buck.Andrej Karpathy — AGI is still a decade away
- The biggest near-term return on compute comes from creative software ideas that squeeze more intelligence out of existing compute units, and Altman believes there are orders of magnitude of efficiency gains still available.Sam Altman - How to Make an Abundant Future - [Invest Like the Best, EP.484]
- For reasoning-model inference, memory (bandwidth and capacity) is the most important of the three GPU axes (FLOPS, memory, interconnect), because the KV cache grows quadratically with context length and reasoning models generate tens of thousands of tokens, creating memory pressure that limits how many users can be served simultaneously.Deepseek Dylan Patel Nathan Lambert
- Output tokens are roughly four times more expensive than input tokens because input (prefill/prompt) can be processed in parallel as a batch, whereas output is autoregressive — each generated token requires re-reading the whole model and KV cache before generating the next, a non-parallel operation.Deepseek Dylan Patel Nathan Lambert
- The time to run an LLM inference can be lower-bounded as the maximum of two terms: the compute time (batch size times active parameters divided by chip FLOPs) and the memory time (total parameters plus KV cache bytes, divided by memory bandwidth); attention compute is small enough to ignore, giving strong predictive power from a simple model.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The dimensionless ratio of compute throughput to memory bandwidth (~300) has remained reasonably stable from A100 to H100 to B100 because, although FLOPs increased substantially across generations, memory bandwidth also increased substantially.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served