multiply-accumulate
An operation that multiplies two numbers and adds the product into a running total; the fundamental primitive at every step of a matrix multiply.
As defined in Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
Where it comes up · 2
- Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up3 claims
- Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]1 claim
Most important claims · 4 of 4Search all claims for “multiply-accumulate” →
- FP4 and FP8 multiply-accumulate circuits are not particularly fungible as drawn, so a chip designer must explicitly choose how much FP4 versus FP8 capacity to include — a decision driven either by customer requirements or by equalizing the power budget between the two precisions.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- In AI chips the accumulation step needs higher precision than the multiplication step because you sum many low-precision products together, causing rounding errors to accumulate quickly, whereas each multiplication involves only one operation with few accumulating errors — which is why a four-bit multiply is paired with an eight-bit addition.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- Multiply-accumulate is the natural fundamental primitive for AI chips because matrix multiplication consists of a triple for-loop where output[i,k] += input[i,j] x input[j,k], so a multiply-accumulate happens at every single step of a matrix multiply.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- The target for a transformer-specific chip is more than 10x better than GPUs, achieved by devoting far more silicon to multiply-accumulate units, since a multiplier takes about 10,000 transistors and NVIDIA devotes only about 4 percent of its chip to them.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]