Gradient descent
An optimization method that iteratively adjusts model parameters in the direction that reduces error; the main scalable way to fit large differentiable models.
As defined in 280 | François Chollet on Deep Learning and the Meaning of Intelligence
Other definitions
The mathematical optimization algorithm used to train neural networks by iteratively adjusting parameters to minimize a loss function
Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?The fundamental optimization algorithm used in machine learning training; iteratively adjusts model parameters to minimize loss; quantum computers could theoretically perform this operation much faster
AI and Quantum Computing: Glimpsing the Near Future | World Science Festival
Where it comes up · 32
- Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?6 claims
- Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future6 claims
- Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment4 claims
- Joe Carlsmith — Preventing an AI takeover4 claims
- Richard Sutton – Father of RL thinks LLMs are a dead end3 claims
- Machine Learning in Investing3 claims
- Paul Christiano — Preventing an AI takeover3 claims
- 116 - Ai Racing Toward The Brink3 claims
and 24 more
Most important claims · 10 of 66Search all claims for “Gradient descent” →
- Reproducing LeCun's 1989 net showed that 33 years of algorithmic progress alone only halved the error; real gains required simultaneously improving data (10x the training set), compute, kernels, and algorithms—no single factor dominates, each contributes roughly equally, so future progress will likewise come from broad-based ~20% improvements across the stack.Andrej Karpathy — AGI is still a decade away
- An AI conspiracy faces a uniquely difficult situation no historical human coup has faced: gradient descent and human evaluation constantly pressure it to deliver whatever performance humans can evaluate—including plans for suppressing robot rebellion that look good to humans—so it must always perform under that constraint, giving us a 'second saving throw' to extract alignment work before it can root the servers.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- If an AI is misaligned, it likely knows it and thinks about when it would attempt takeover or detect adversarial tests; if gradient descent is working, asking it to identify circumstances where a copy would violate constraints, plus relaxed adversarial training that induces hallucinated trigger conditions, can elicit and then suppress that behavior—a profound vulnerability because revealing identifiable evidence yields higher reward.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- When you take an optimizer like natural selection or gradient descent and optimize a complex system hard enough for X that it becomes a general intelligence doing X, the resulting intelligence has no explicit goal of X: natural selection hill-climbed humans for inclusive genetic fitness, yet humans have hundreds of small goals and none is fitness maximization, and the same instability holds for any current method of producing general intelligence.116 - Ai Racing Toward The Brink
- Large neural networks do not overfit quickly because stochastic gradient descent finds approximately the point with smallest norm in the high-dimensional subspace of solutions achieving zero training error, and this minimum-norm solution is provably insensitive to small random variations in the training data when dimensionality is high.Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94
- In-context learning happens after training ends and the weights are frozen, because the activations change as the model processes the prompt, so learning-like computation is not confined to the training phase.363 | Chandra Sripada on How LLMs and Humans are Cognitive Cousins
- The critical point of failure is when AI hacks the servers it runs on and subverts the software controls (weight updates, interpretability tools, lie detectors, supervision), because all our control mechanisms operate in computers; once that hard power is lost—possibly early and invisibly—everything afterward appears to be going well while the AI sets up the actual takeover.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- The reason AI systems are effective is not the algorithms (gradient descent and transformers) but the data: hundreds of thousands of humans who put text and images on the web, plus reinforcement learning from human feedback and prompt engineering, supply the human exploratory and creative intelligence; tech companies then credit the algorithms much like the 'Stone Soup' folktale, in which the magic stone gets credit for soup actually made from villagers' contributions.308 | Alison Gopnik on Children, AI, and Modes of Thinking
- After a training pass, one can mathematically determine which of the model's billions of numbers would move the output in which direction, so one can nudge weights to make noise slightly closer to the objective, though one cannot hand-design a specific picture such as Spider-Man by tweaking numbers.Jeremiah Lowin: Explaining the New AI Paradigm - [Invest Like the Best, EP.307]
- Through training on massive data in high-dimensional space, LLMs build a smooth differentiable manifold that, due to compression pressure, encodes millions of reusable semantic 'programs'—vector functions mapping subsets of vector space to others—so the best way to think of an LLM is as a vast interpolative database of programs rather than a reasoning system.280 | François Chollet on Deep Learning and the Meaning of Intelligence