Sheela
Sheela
Home
Add
Questions
Thinkers
Claims
People
Concepts
Jargon
Sources
Health
Predictions
Collections
Domain Landscape
Contribute
Review
Dedup
Topic requests
Settings
Help

← Jargon

Jargontechnology· 2 claims· 2 episodes

distributed computing

Coordinating computation across many machines or data centers, a challenge in training very large models.

As defined in Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold


Where it comes up · 2
  • Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold1 claim
  • Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]1 claim

Most important claims · 2 of 2Search all claims for “distributed computing” →
  • Around 2005 CPU speeds stagnated at the so-called Moore's Wall, so computers stopped getting faster and computation and data moved into data centers with hundreds or thousands of machines, because demand for processing kept growing exponentially.Ali Ghodsi – The Past, Present, and Future of Big Data – [Founder’s Field Guide, EP.18]
  • Scaling laws don't work by magic; at each new scale you must adjust hyperparameters and add innovations rather than repeat the same recipe, because extrapolating predictions several orders of magnitude can fail, capabilities can appear as step functions, and intermediate data points are needed to keep the scaling law true—making scaling an art form, with about one order of magnitude the maximum jump between eras.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold