HBM
High Bandwidth Memory—stacked memory offering very high bandwidth, which Huang convinced DRAM makers would become mainstream for data centers.
As defined in Lex Fridman Podcast #494 - Jensen Huang
Where it comes up · 17
- Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]7 claims
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served6 claims
- Dwarkesh Patel Podcast - Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 20284 claims
- Dwarkesh Patel Podcast - Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute3 claims
- @Asianometry & Dylan Patel — How the semiconductor industry actually works2 claims
- Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI2 claims
- Dwarkesh Patel Podcast - Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat2 claims
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]2 claims
and 9 more
Most important claims · 10 of 37Search all claims for “HBM” →
- On chips like the H100 or AMD MI300X, memory costs more than the silicon itself by more than a factor of two, and memory is made by only two or three firms, Samsung, SK Hynix, and possibly Micron, making them critical supply chain nodes.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- There is a misconception that Mixture of Experts means you avoid retrieving unused expert weights; in reality, because efficient serving uses very large batches of independent requests, all experts are looked at every step, with only a small fraction of the batch sent through each expert—so you still need the whole model in memory (enough HBM to hold it), and running an unused expert at batch size one would be massively inefficient given modern hardware's high operational intensity.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- Doubling every input would not solve the problem because bottlenecks are balanced across the system, so fixing one, such as a TSMC constraint, merely exposes the next bottleneck immediately.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- The biggest source of non-deterministic latency in a CPU is the cache, which transparently decides in hardware whether data comes from fast cache or slow DDR — a decision that depends on ambient conditions like other running programs and the cache's random replacement, making runtime unpredictable; TPUs instead use a scratchpad model where separate explicit instructions target scratchpad versus HBM, baking the decision into software for deterministic latency.Dwarkesh Patel Podcast - Reiner Pope – Chip design from the bottom up
- A key limit on AI performance is the memory wall: data must move between processor and memory at speeds limited by electrons, so architectures that bring memory closer, or use optical interconnects, could lower the cost of compute and reshape who leads in chip production.Andrew Homan & Chris Miller - Redefining Semiconductor Progress - [Invest Like the Best, EP.396]
- KV cache can be regenerated from scratch (rematerialization, costing a forward pass) or stored across memory tiers (HBM, DDR, flash, spinning disk); the optimal tier balances retrieval cost against holding cost, with the rule that you pick a memory whose drain time (capacity divided by bandwidth) matches how long you intend to hold the data—so cache pricing tiers (e.g., 5-minute vs 1-hour) likely correspond to specific physical memory tiers.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- About three years ago Huang convinced several DRAM CEOs to invest in HBM memory for data centers when it was used scarcely, and to adapt low-power cell-phone memory (LPDDR) for supercomputers, despite their initial disbelief; all three memory companies then had record years, demonstrating his role in shaping the supply chain's future.Lex Fridman Podcast #494 - Jensen Huang
- Hardware companies cannot control their full stack as software companies can, because they depend on HBM, DRAM, TSMC, and cross-border supply chains, and Cerebras' 2024 IPO attempt failed over CFIUS and timing, so getting lucky on timing and macro conditions is critical.Eric Vishria - A Decade of Lessons Investing in Software & Hardware - [Invest Like the Best, EP.486]
- Larger scale-up domains massively reduce weight memory-fetch latency because all GPUs in a scale-up domain load weights in parallel, so effective bandwidth equals scale-up size times per-GPU bandwidth; this term grew ~8x from Hopper while per-GPU bandwidth only grows ~1.5-2x per generation, meaning scale-up size—not memory capacity—solves the bandwidth/latency problem.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- The ~20 millisecond batch interval comes from wanting to read all of HBM capacity into the chip exactly once per forward pass—capacity divided by bandwidth—which is about 20 milliseconds across many HBM generations (e.g., Rubin's ~288 GB over ~20 TB/s ≈ 15 ms); running slower would wastefully re-read read-only weights and KVs.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served