factual

Inference scheduled as fixed-interval departing trains

Inference runs on a fixed schedule like a departing train: a new batch starts every ~20 milliseconds regardless of fullness; ready requests board, full trains leave waiting passengers for the next one, so worst-case queuing latency is one train interval plus one completion—about 40 milliseconds.

factualpending

Speaker

Reiner Pope

Evidence Quote

“A new train departs every 20 milliseconds. Any passengers who are ready board the train. If the train is full, they wait until the next train. If the train is not full, the train is going to go anyway.”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...