factual
Google already does synchronous multi-datacenter training
Google already trains models across multiple metro areas, as disclosed in the Gemini 1.5 tech report, using high-bandwidth but high-latency connections between data centers; this works because each training step takes seconds, so 50-millisecond latency is irrelevant—only bandwidth matters, as long as parameters can be synced and gradients accumulated across data centers within the step time.
factualpending
Speaker
Jeff DeanEvidence Quote
“Training is kind of interesting because each step in a training process is usually, for a large model, is usually a few seconds or something, at least. So, the latency of it being 50 milliseconds away doesn't matter that much.”
Created: 6/13/2026, 3:30:41 AM
Your notes
Loading notes...