distributed computing
Coordinating computation across many machines or data centers, a challenge in training very large models.
As defined in Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
Where it comes up · 2
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold1 claim
- Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]1 claim
Most important claims · 2 of 2Search all claims for “distributed computing” →
- Around 2005 CPU speeds stagnated at the so-called Moore's Wall, so computers stopped getting faster and computation and data moved into data centers with hundreds or thousands of machines, because demand for processing kept growing exponentially.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
- Scaling laws don't work by magic; at each new scale you must adjust hyperparameters and add innovations rather than repeat the same recipe, because extrapolating predictions several orders of magnitude can fail, capabilities can appear as step functions, and intermediate data points are needed to keep the scaling law true—making scaling an art form, with about one order of magnitude the maximum jump between eras.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold