causal

Arithmetic area scales quadratically with bit width

Because multiplier circuit area scales quadratically (p×q) with bit width, halving the precision should more than quadruple — not merely double — the achievable FLOP count, which is the single biggest reason low-precision arithmetic has worked so well for neural nets; Nvidia historically doubled FLOPs per halved precision but only acknowledged the larger speedup (3x FP4 vs FP8) in B300 and beyond, though it arguably should be 4x.

causalpending

Speaker

Reiner Pope

Evidence Quote

“This is a really big reason... because of this quadratic scaling, that ratio is actually slightly wrong. You should get an even bigger speedup than you might otherwise think.”

Source

Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:33:43 AM

Your notes

Loading notes...