factual
Memory tier choice balances retrieval and holding cost
KV cache can be regenerated from scratch (rematerialization, costing a forward pass) or stored across memory tiers (HBM, DDR, flash, spinning disk); the optimal tier balances retrieval cost against holding cost, with the rule that you pick a memory whose drain time (capacity divided by bandwidth) matches how long you intend to hold the data—so cache pricing tiers (e.g., 5-minute vs 1-hour) likely correspond to specific physical memory tiers.
factualpending
Speaker
Reiner PopeEvidence Quote
“given that I know I’m going to be holding something for five minutes, I would like to pick a memory that I can read every five minutes... if I take the storage capacity over storage bandwidth, I would like this to be equal to five minutes.”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...