SRAM
Static random-access memory; on-chip memory that no longer shrinks with newer process nodes.
As defined in @Asianometry & Dylan Patel — How the semiconductor industry actually works
Where it comes up · 6
- Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]5 claims
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]2 claims
- AI Market Jitters1 claim
- @Asianometry & Dylan Patel — How the semiconductor industry actually works1 claim
- Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up1 claim
- Eric Vishria - A Decade of Lessons Investing in Software & Hardware - [Invest Like the Best, EP.486]1 claim
Most important claims · 10 of 11Search all claims for “SRAM” →
- There are three known ways to speed up deep learning in hardware: increase the number of cores, increase communication between cores, and bring memory closer to compute, and Cerebras pushed all three to the extreme with a wafer-scale chip holding about 450,000 cores and about 20 GB of on-chip SRAM.Eric Vishria - A Decade of Lessons Investing in Software & Hardware - [Invest Like the Best, EP.486]
- On-chip SRAM is roughly three orders of magnitude less dense than DRAM-based HBM; Blackwell carries 288 gigabytes of HBM around its logic die against about 500 megabytes of on-die SRAM, though SRAM offers much faster access.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Chip design revolves around a memory hierarchy trade-off: off-chip memory gives capacity while on-chip memory gives speed, so hybridizing the two is necessary for models that need both large weights and dynamic context.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Wafer-scale SRAM architectures can host a trillion-parameter model's weights and reach about 1,000 tokens per second because weights stream in about a millisecond, but the growing, dynamic KV cache is a persistent limit those chips handle poorly.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Cerebras bets on wafer-scale integration with around 50 gigabytes of SRAM per wafer, stacked to about a terabyte and offering roughly 21 petabytes per second of bandwidth, enough to move a trillion-parameter model like Kimi in about a millisecond and reach around 1,000 tokens per second.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- SRAM-based accelerators not constrained by HBM DRAM and built on older nodes can handle the feedforward network in disaggregated inference, which Gavin argues will improve AI ROI by splitting prefill, attention, and feedforward work across specialized chips.AI Market Jitters
- MatX has publicly discussed a 'splittable systolic array' — big systolic arrays that can also act as small systolic arrays — as a way to capture the GPU-like benefit of smaller flexible structures surrounded by SRAM while discarding the space-consuming CUDA-architecture support hardware that GPUs require in each SM.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- Cerebras- and Groq-style SRAM-heavy accelerators are best used alongside GPUs with off-chip memory, because on-chip SRAM gives speed for weights while off-chip memory gives capacity for KV cache; Nvidia and Groq may be doing exactly this.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- For decode, performance depends on memory bandwidth across the full scale-up cluster rather than on a single chip, because loading the model and KV cache faster allows more tokens per second per user, so the right question is cluster bandwidth rather than chip bandwidth.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- Moving a process node no longer doubles density or power efficiency (Dennard scaling ended; SRAM doesn't scale, logic scales ~30%, only ~20% power saving per transistor), but AI gains far more than the per-transistor benefit because higher density lets you fit more of a computation (e.g., a large matrix multiply) on-chip, reducing costly off-chip data movement to memory - so data locality, not transistor power, drives AI's hunger for new nodes.@Asianometry & Dylan Patel — How the semiconductor industry actually works