causal

Cache non-determinism vs scratchpad determinism

The biggest source of non-deterministic latency in a CPU is the cache, which transparently decides in hardware whether data comes from fast cache or slow DDR — a decision that depends on ambient conditions like other running programs and the cache's random replacement, making runtime unpredictable; TPUs instead use a scratchpad model where separate explicit instructions target scratchpad versus HBM, baking the decision into software for deterministic latency.

causalpending

Speaker

Reiner Pope

Evidence Quote

“whether or not you get a cache hit depends on the ambient environment of the CPU: what other programs are running, what has run recently, and what the random number generator inside the cache system is doing. That is a big source of non-determinism in the runtime of a CPU.”

Source

Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:33:43 AM

Your notes

Loading notes...