systolic array
A grid of multiply-accumulate cells that stores a weight matrix locally and streams vectors through it, the most efficient known circuit for matrix multiplication; generic name for Tensor Cores.
As defined in Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
Where it comes up · 2
- Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up7 claims
- Dwarkesh Patel Podcast - Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat1 claim
Most important claims · 8 of 8Search all claims for “systolic array” →
- At the top level a GPU is a regular grid of near-identical SMs around an L2 memory, while a TPU is a few coarse-grained matrix units (large systolic arrays) with a vector unit in the middle; since an SM is essentially a scaled-down matrix unit plus small vector unit, a GPU can be understood as a lot of tiny TPUs tiled across the whole chip.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- Because the weight matrix changes only rarely, it can be loaded into the systolic array by trickle-feeding it slowly through a daisy chain — bringing in one row per clock cycle and shifting other rows down — which bounds the wiring that crosses the array boundary to x rather than x×y; this exploits the distinction between communication bandwidth (which costs die area) and communication time.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- A systolic array bakes two levels of the matrix-multiply loop into hardware so that the weight matrix is stored locally in registers and reused across many input vectors; this provides x×y as much compute while keeping input/output communication at only x, solving the data-movement problem that plagued CUDA cores and making it the most efficient known circuit for matrix multiplication.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- The TPU's coarse-grained design allows larger systolic arrays that amortize register-file costs better, but it constrains data movement between vector and matrix units to just two lines of perimeter, whereas a GPU's many small units let data move through many more lines (e.g. 16); GPUs thus achieve higher vector-to-matrix bandwidth and cheaper intra-SM data movement, but cross-SM operation becomes complicated and expensive.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- Most chip design decisions are sizing decisions — for example, how big to make the systolic array versus the register file — which are coupled and can be approached by setting a budget for what percentage of chip area is spent on data movement (e.g. 10% register file, 90% systolic array), trading register-file flexibility and application performance against compute area.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- MatX has publicly discussed a 'splittable systolic array' — big systolic arrays that can also act as small systolic arrays — as a way to capture the GPU-like benefit of smaller flexible structures surrounded by SRAM while discarding the space-consuming CUDA-architecture support hardware that GPUs require in each SM.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- Because Moore's Law now yields only ~25% per year, the only way to achieve 10x-100x leaps is to change the algorithm and how it's computed yearly; Nvidia's general programmability via CUDA enables new architectures (hybrid SSMs, fused diffusion/autoregressive models, MoEs), which is why Blackwell achieved roughly 50x Hopper's efficiency despite only ~75% transistor improvement.Dwarkesh Patel Podcast - Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat
- Although the brain is often distinguished from chips by co-locating memory and compute, modern chips also co-locate memory and compute (e.g. the systolic array storing weights locally next to the logic), so this is less of a fundamental difference than commonly assumed; the more significant differences are unstructured sparsity in the brain and its much slower clock cycle.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up