MFU
Model FLOPs Utilization; the fraction of a chip's peak compute actually used during a workload.
As defined in Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
Where it comes up · 3
- Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]11 claims
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]3 claims
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served2 claims
Most important claims · 10 of 16Search all claims for “MFU” →
- Model FLOPS utilization (MFU), the share of theoretical compute actually applied to training, runs around 35 to 40 percent, with published GPT-3, LaMDA and Megatron figures in the high 20s to high 30s, and Baker argues it is the best single metric for evaluating a lab.Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]
- Utilization cannot reach 100 percent because increasing switching activity draws more power and heat, so the chip self-regulates and lowers clock speed; adding more FLOPS without solving thermals therefore yields no extra performance.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- Model FLOPS utilization (MFU) measures how much of the advertised peak FLOPS a chip actually delivers on real workloads, and GPUs often achieve only 20 to 50 percent depending on the workload, so the relevant metric is achieved rather than headline FLOPS.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- RL training is less efficient per token than pre-training for two reasons: you may not train a backward pass on all rollouts (so FLOPs per token can be as low as 2 rather than the full forward+backward 6), and RL involves substantial decode which often runs at lower MFU than training, together giving roughly a 30% efficiency penalty.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- When memory fetch is linear in context length (true for dense attention), running at twice the optimal context length can roughly halve model FLOPs utilization, because you push from the balanced compute-bound/memory-bound point into being memory-limited; sparse attention scales much better and avoids this.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- MFU decomposes into MAMMF (software efficiency) times SFU (system FLOPS efficiency) times the share of time not spent checkpointing, and then times PUE (power utilization efficiency), so these factors compound multiplicatively into the final efficiency of a training system.Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]
- Compounding moderate efficiency gaps produces large output differences: a firm at 90 percent MAMMF, 50 percent SFU and checkpointing one-third of the time reaches about 30 percent MFU, while a firm at 100 percent MAMMF, 60 percent SFU and 10 percent checkpointing reaches about 54 percent.Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]
- A higher MFU lets a lab choose among faster time to market (45 percent faster at 50 versus 40 percent MFU), higher quality, or lower cost; a 25 percent higher-quality model combined with quantization can cut inference cost by almost 50 percent.Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]
- Over the last five years GPUs have gotten roughly 50x faster while the rest of the data center (networking, storage, memory) got only 4x to 5x faster, so GPUs sit idle waiting for data and MFU stays low.Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]
- Frontier models will likely converge to similar intelligence unless a lab builds a compounding advantage in MFU, SFU, checkpointing, and PUE, and firms like Google and Meta will try to close gaps by spending more money.Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]