factual

Google already does synchronous multi-datacenter training

Google already trains models across multiple metro areas, as disclosed in the Gemini 1.5 tech report, using high-bandwidth but high-latency connections between data centers; this works because each training step takes seconds, so 50-millisecond latency is irrelevant—only bandwidth matters, as long as parameters can be synced and gradients accumulated across data centers within the step time.

factualpending

Speaker

Jeff Dean

Evidence Quote

“Training is kind of interesting because each step in a training process is usually, for a large model, is usually a few seconds or something, at least. So, the latency of it being 50 milliseconds away doesn't matter that much.”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...