factual

Memory tier choice balances retrieval and holding cost

KV cache can be regenerated from scratch (rematerialization, costing a forward pass) or stored across memory tiers (HBM, DDR, flash, spinning disk); the optimal tier balances retrieval cost against holding cost, with the rule that you pick a memory whose drain time (capacity divided by bandwidth) matches how long you intend to hold the data—so cache pricing tiers (e.g., 5-minute vs 1-hour) likely correspond to specific physical memory tiers.

factualpending

Speaker

Reiner Pope

Evidence Quote

“given that I know I’m going to be holding something for five minutes, I would like to pick a memory that I can read every five minutes... if I take the storage capacity over storage bandwidth, I would like this to be equal to five minutes.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...