Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
What this covers
Reiner Pope walks through the physics and circuit design of modern AI chips, arguing that a single principle — maximizing useful computation relative to data movement overhead — explains the structural choices across all processor types. Dwarkesh Patel hosts. The conversation moves systematically from fundamental arithmetic (how bit-width scales with area) through circuit primitives (multiply-accumulate, mux, register file) to the macro-level design differences between CPU cores, GPU SMs, systolic arrays, and FPGAs. Pope uses the compute vs communication trade-off as a lens: a naive CUDA core wastes roughly seven-eighths of its area moving data to and from the ALU rather than computing; systolic arrays solve this by storing weights locally and reusing them across many inputs; low-precision arithmetic works because halving bit-width should quadruple throughput (since multiplier area scales with the square of precision), though Nvidia only recently acknowledged speedups beyond the naive doubling expected from each precision halving.
The conversation covers distinct design spaces and the trade-offs that govern them. Precision choice is hardware-baked — a chip designer must decide how much FP4 versus FP8 capacity to build in. Pipeline register insertion is a pure exchange between clock speed and silicon area, but over-pipelining becomes self-defeating: too many registers for synchronization leave no room for actual logic. FPGAs cost roughly ten times more than ASICs (the ten-to-one cost penalty comes from lookup tables that enumerate truth tables rather than placing gates), which makes them viable only when you need reprogrammability and can absorb the overhead — deterministic latency matters more than raw efficiency. CPU cores are roughly 100 times larger than GPU cores; Pope attributes much of this to the branch predictor, a circuit with no GPU equivalent. The latency vs throughput trade-off surfaces repeatedly: running a chip 1000 times slower cuts energy consumption by 1000 times, but not energy efficiency, so the brain's efficiency cannot be replicated by mere slowness. Clock speed is ultimately bound by combinational logic depth and the feedback loops (accumulators, running sums) that cannot be naively pipelined without changing the computation.
Pope argues that AI chip design is fundamentally a battle to maximize useful compute relative to data-movement/communication overhead, and that this single trade-off explains why low-precision arithmetic, systolic arrays, scratchpads, and the structural differences between GPUs, TPUs, FPGAs, CPUs, and brains all look the way they do.
- In a naive CUDA/CPU core, ~7/8 of circuit area is spent moving data from register file to ALU rather than on the multiply itself
- Systolic arrays store the weight matrix locally to raise compute-per-communication, amortizing register-file cost as the array grows
- Precision scales quadratically in area, so halving bit-width should more than double throughput
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Because multiplier circuit area scales quadratically (p×q) with bit width, halving the precision should more than quadruple — not merely double — the achievable FLOP count, which is the single biggest reason low-precision arithmetic has worked so well for neural nets; Nvidia historically doubled FLOPs per halved precision but only acknowledged the larger speedup (3x FP4 vs FP8) in B300 and beyond, though it arguably should be 4x.
“the amount of area it takes sounds like it’s quadratic with the bit length. That’s why smaller precision is even more favorable than you might think.”
Running a chip 1000x slower (megahertz instead of gigahertz) gives roughly 1000x fewer transitions and thus ~1000x less energy consumption — since most chip energy is the dynamic/switching power of charging and discharging capacitors when bits toggle — but it does not produce a substantial gain in energy efficiency, so silicon cannot replicate the brain's efficiency merely by slowing down.
“If you run a chip much slower and you only clock it once every thousand clock cycles, you will have 1,000 times fewer transitions. It will be about 1,000 times less energy consumption. But it’s not a substantial advantage in energy efficiency.”
Anything expressible in an FPGA can be expressed in an ASIC at roughly an order of magnitude lower cost and better energy efficiency, but the first FPGA costs ~$10,000 while the first ASIC costs ~$30 million due to tape-out; so FPGAs make business sense when you need deterministic latency, fast runtime, and high parallelism but will change the workload frequently (e.g. monthly) and don't want to repay the tape-out cost.
“It will be about an order of magnitude cheaper and have better energy efficiency on an ASIC than an FPGA.”
FP4 and FP8 multiply-accumulate circuits are not particularly fungible as drawn, so a chip designer must explicitly choose how much FP4 versus FP8 capacity to include — a decision driven either by customer requirements or by equalizing the power budget between the two precisions.
“As drawn, they’re not particularly fungible. This is actually one of the main choices you have to make when designing a chip: how much of FP4 and how much of FP8 do I have?”
At the top level a GPU is a regular grid of near-identical SMs around an L2 memory, while a TPU is a few coarse-grained matrix units (large systolic arrays) with a vector unit in the middle; since an SM is essentially a scaled-down matrix unit plus small vector unit, a GPU can be understood as a lot of tiny TPUs tiled across the whole chip.
“From a very high-level point of view, the GPU has a lot of tiny TPUs tiled across the whole chip.”
The biggest source of non-deterministic latency in a CPU is the cache, which transparently decides in hardware whether data comes from fast cache or slow DDR — a decision that depends on ambient conditions like other running programs and the cache's random replacement, making runtime unpredictable; TPUs instead use a scratchpad model where separate explicit instructions target scratchpad versus HBM, baking the decision into software for deterministic latency.
“probably the most important source of non-determinism on a CPU is the CPU cache itself.”
A systolic array bakes two levels of the matrix-multiply loop into hardware so that the weight matrix is stored locally in registers and reused across many input vectors; this provides x×y as much compute while keeping input/output communication at only x, solving the data-movement problem that plagued CUDA cores and making it the most efficient known circuit for matrix multiplication.
“The idea of a systolic array is to go two levels of loops up and bake this entire loop out here into hardware. The idea is that if we have a much bigger granularity fixed-function piece of logic, maybe the taxes we pay on the input and output are much smaller.”
A Dadda multiplier built from full adders (3→2 compressors) uses exactly p×q full adders for a p-bit by q-bit multiply-accumulate, because the input bits (p×q partial products plus p+q accumulator bits) minus the p+q output bits equals p×q, and each full adder eliminates exactly one bit.
“This approach is called a Dadda multiplier. This is the standard for how you do area-efficient multipliers using full adders.”
Because the weight matrix changes only rarely, it can be loaded into the systolic array by trickle-feeding it slowly through a daisy chain — bringing in one row per clock cycle and shifting other rows down — which bounds the wiring that crosses the array boundary to x rather than x×y; this exploits the distinction between communication bandwidth (which costs die area) and communication time.
“since we’re only going to be loading this in once, let’s minimize bandwidth, because bandwidth equals die area. We load it in slowly over smaller lanes because we’re just going to keep this value in there for a while.”
Driving clock speed very high by inserting many pipeline registers (e.g. an AND gate plus register in a tight loop hitting 5-6 GHz) is counterproductive because almost all area then goes to synchronization/registers rather than logic; throughput equals work-per-clock (area efficiency) times clocks-per-second, so over-pipelining gives low latency but low throughput.
“the throughput of your chip is the product of how much you get done per clock cycle—which is based on area efficiency—times how many clocks you get per second.”
Logic that feeds back on itself (a loop, such as a running sum that adds a number every clock cycle) cannot be naively pipelined, because inserting a register in the middle changes the computation — splitting it into separate sums of even and odd inputs — so these loops, which all chips have somewhere, are the hardest constraint and ultimately set the clock cycle.
“This constraint—where I have a loop in my logic, which all chips have somewhere—is the hardest thing to address and sets the clock cycle.”
The roughly 10x cost penalty of an FPGA versus an ASIC comes from the LUT: a four-way AND that takes three AND gates in an ASIC requires a 32-gate LUT (a 16-input, 1-bit mux) to implement, because a truth table lists every input combination whereas placing actual gates and wires is far more concise.
“This is a circuit I could implement in an ASIC directly using three AND gates. Using a LUT, I can also implement it, but it’s going to take 32 gates instead of three.”
CPU cores are ~100x bigger than GPU cores largely because of the branch predictor — a large area dedicated to predicting where and when branches occur — which has no equivalent in a GPU; stripping out the branch predictor and tightening the register files drives much of the GPU's density advantage over the CPU.
“the thing that does not have an equivalent in a GPU is the branch predictor. There is a whole big area in the CPU which is just a bunch of predictors... Stripping a lot of that out, as well as making these register files tighter, drives a lot of the GPU gains over the CPU.”
Selecting one input out of n in hardware — invisible to the software programmer who just says 'I want register three' — actually requires a mux costing n×p AND gates and (n−1)×p OR gates, built by masking every entry and ORing the results together.
“you don’t think at the level of hardware. You just think, “Oh, I’ll just select element three,” and something as simple as that is in and of itself quite a complicated circuit.”
The TPU's coarse-grained design allows larger systolic arrays that amortize register-file costs better, but it constrains data movement between vector and matrix units to just two lines of perimeter, whereas a GPU's many small units let data move through many more lines (e.g. 16); GPUs thus achieve higher vector-to-matrix bandwidth and cheaper intra-SM data movement, but cross-SM operation becomes complicated and expensive.
“The amount of data you can move between a vector unit and a matrix unit is actually much higher in a GPU than in a TPU. Instead of having to move all the data through just two lines, you’re moving it through 16 lines of wiring in a GPU.”
In AI chips the accumulation step needs higher precision than the multiplication step because you sum many low-precision products together, causing rounding errors to accumulate quickly, whereas each multiplication involves only one operation with few accumulating errors — which is why a four-bit multiply is paired with an eight-bit addition.
“the precision will almost always be higher in the accumulation step than in the multiplication step. This is specific to AI chips. You’re multiplying low-precision numbers, and then when you accumulate, errors accumulate quickly, so you need more precision there.”
In a pre-Tensor-Core CUDA/CPU core, the cost of moving data from the register file to the ALU and back vastly exceeds the cost of the logic unit itself: with three muxes (one per multiply-accumulate input) over an 8-entry register file, roughly seven-eighths of the circuit area is spent on reading and writing registers and only a tiny fraction on the actual computation.
“all of this work just moving the data from the register file to the logic unit is many, many times more expensive than the logic unit.”
A chip's maximum clock speed is bounded by the propagation delay through each cloud of combinational logic, since any computation must finish before the next clock edge; chip designers margin this so the result settles ~25% of the clock cycle in advance, making timing failure many standard deviations unlikely (except at clock domain crossings).
“I’m very sensitive to the delay through this cloud of logic, because any computation happening in there needs to finish before the next clock cycle hits.”
Multiply-accumulate is the natural fundamental primitive for AI chips because matrix multiplication consists of a triple for-loop where output[i,k] += input[i,j] x input[j,k], so a multiply-accumulate happens at every single step of a matrix multiply.
“There’s a for-loop over i, over j, and over k, of output [i,k] += input [i,j] x other input [j,k]. A multiply-accumulate happens at every single step of a matrix multiply.”
A branch predictor exists because evaluating a branch — noticing it, evaluating the Boolean, updating the program counter, and reading from instruction memory — can take ~5 nanoseconds (a 200 MHz limit), but designers want to run at 1-2 GHz, so the predictor guesses the branch outcome about five cycles early and keeps running subsequent instructions, rolling back if the prediction was wrong.
“The purpose of the branch predictor is to predict, five cycles earlier, that a branch is going to happen, before you even get to that instruction.”
Unlike software which uses expensive synchronization like mutexes, chips synchronize their ~100 billion transistors by having all circuitry pause roughly every nanosecond and advance in lockstep on a global clock signal mediated by registers, which store whatever value is on their input wire at the instant the clock strikes.
“Every nanosecond or so, all circuitry in the chip will pause for a moment and synchronize. That is the clock cycle. The entire chip typically goes in lockstep to the next operation in one fell swoop.”
Splitting a cloud of logic in half with a pipeline register lets you hit roughly twice the clock frequency at the cost of an extra register (more storage); inserting these registers is a huge, semi-automated part of chip design and is a pure trade-off between clock speed and area.
“If you split it in the middle, you can hit twice the clock frequency. That’s great, you get twice the performance, but at the cost of an extra register, which means more storage.”
An FPGA emulates the ASIC programming model in fixed hardware using registers for storage, lookup tables (LUTs) that act as programmable gates, and a large set of muxes in front of every component that select inputs from elsewhere; programming the FPGA 'in the field' amounts to configuring the control bits of every mux to superimpose a particular wiring through the LUTs and registers.
“At its core, it has the two components we just talked about. It has registers as storage devices, and it has lookup tables (LUTs) which provide all of the gates.”
A lookup table with four bits of input and one bit of output stores a 16-entry table (the truth table) selected by the input bits, letting it configurably take the role of any gate (AND, OR, XOR, etc.); four inputs is the standard sweet spot because too few inputs requires more LUTs — itself a compute-vs-communication trade-off.
“A lookup table has four bits of input and one bit of output. How many different functions are there from four bits to one bit? There are 16 different functions.”
The CPU cache is about two orders of magnitude faster than DDR memory, so without it basically all programs would run roughly a hundred times slower, making the cache absolutely necessary for a CPU to run at reasonable speed.
“The cache is two orders of magnitude faster than the DDR. If you never used the cache, basically all programs would run a hundred times slower.”
Two chips manufactured on the same process node (e.g. 3nm at TSMC) can have different clock cycles because of manufacturing variance in how long signals take through different logic paths, and because optimization to ensure no single critical path is so long that it slows the whole chip's clock differs between designs.
“This explains why different chips made at the same process node, the same TSMC technology, can have different clock cycles... based on whether they were able to optimize to ensure no single critical path is so long that it slows down the whole chip’s clock cycle.”
Most chip design decisions are sizing decisions — for example, how big to make the systolic array versus the register file — which are coupled and can be approached by setting a budget for what percentage of chip area is spent on data movement (e.g. 10% register file, 90% systolic array), trading register-file flexibility and application performance against compute area.
“Most of the decisions in chip design are sizing decisions.”
MatX has publicly discussed a 'splittable systolic array' — big systolic arrays that can also act as small systolic arrays — as a way to capture the GPU-like benefit of smaller flexible structures surrounded by SRAM while discarding the space-consuming CUDA-architecture support hardware that GPUs require in each SM.
“We’ve talked publicly about something we call a splittable systolic array, which in some sense you can think of as big systolic arrays that can be small systolic arrays too.”
Although the brain is often distinguished from chips by co-locating memory and compute, modern chips also co-locate memory and compute (e.g. the systolic array storing weights locally next to the logic), so this is less of a fundamental difference than commonly assumed; the more significant differences are unstructured sparsity in the brain and its much slower clock cycle.
“This is exactly the co-location, in some sense, of the memory and compute.”
An AND gate is almost the simplest and smallest logic gate on a chip, while a full adder is typically the largest logic gate used; a full adder adds three single-bit numbers and outputs two bits, also known as a 3→2 compressor.
“AND is almost the simplest logic gate that exists on a chip. It’s almost the smallest. At the other extreme, the very largest logic gate you’ll typically use is something called a full adder.”
At the very bottom level of a chip, the primitives are logic gates like AND, OR, and NOT, which are connected by wires laid out physically as metal traces on the chip.
“At the very bottom level of a chip, the primitives we work with are logic gates, very simple things like AND, OR, and NOT. These are connected together by wires that have to be laid out physically as metal traces on a chip.”