factual
Synchronous training is easier to reason about
Google moved from asynchronous CPU training to fully synchronous TPU training because synchronous runs are replicable—results no longer depend on incidental factors like a web crawler running on the same machine—making it an easier mental model where only your algorithm fights you rather than asynchrony plus the algorithm both battling you, though at greater scale some asynchrony may return.
factualpending
Speaker
Noam ShazeerEvidence Quote
“it is an easier mental model to understand. You just have your algorithm sort of fighting you, rather than the asynchrony and the algorithm kind of battling you.”
Created: 6/13/2026, 3:30:41 AM
Your notes
Loading notes...