factual

Synchronous training is easier to reason about

Google moved from asynchronous CPU training to fully synchronous TPU training because synchronous runs are replicable—results no longer depend on incidental factors like a web crawler running on the same machine—making it an easier mental model where only your algorithm fights you rather than asynchrony plus the algorithm both battling you, though at greater scale some asynchrony may return.

factualpending

Speaker

Noam Shazeer

Evidence Quote

“it is an easier mental model to understand. You just have your algorithm sort of fighting you, rather than the asynchrony and the algorithm kind of battling you.”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...