Backpropagation
The technical method by which learning is applied in neural networks, adjusting weights based on output errors; the core algorithm behind deep learning, developed in the 1980s.
As defined in et_648
Other definitions
Training algorithm (still in use today for deep learning and large language models) that adjusts neural network weights by measuring error and propagating corrections backward through layers
AI: Grappling with a New Kind of Intelligence | World Science Festival
Where it comes up · 23
- The Brain’s Learning Algorithm Isn’t Backpropagation4 claims
- 336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI4 claims
- Ilya Sutskever: Deep Learning | Lex Fridman Podcast #943 claims
- Geoff Hinton - Will Digital Intelligence Replace Biological Intelligence? | Vector's Remarkable 20243 claims
- History of Science and Technology Q&A (February 8, 2023)2 claims
- Finally an NVIDIA Competitor for AI? Maybe, RAIN's Brain chips!!!2 claims
- Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence2 claims
- Geoffrey Hinton - Two Paths to Intelligence2 claims
and 15 more
Most important claims · 10 of 37Search all claims for “Backpropagation” →
- The key piece of math that made it possible to train multilayer neural networks—the backpropagation algorithm developed by Rumelhart and Hinton—relies on the chain rule of calculus, which traces back to Leibniz, closing the historical loop in which Leibniz's mathematics underpins the modern deep-learning revolution.343 | Tom Griffiths on The Laws of Thought
- The 1986 Rumelhart-Hinton-Williams backpropagation paper showed that multi-layer neural networks can be trained using what turns out to be just the chain rule of calculus, which requires that the entire computational graph from output to input be differentiable—achieved by replacing the non-differentiable step threshold with a smooth sigmoid.336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
- Bernard Widrow's adaptive digital filter work led to the Widrow-Hoff Least Mean Square algorithm, which is the true precursor to today's backpropagation; it was effectively a very noisy, algebraic version of stochastic gradient descent that avoided calculus so it could be implemented directly in hardware.336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
- Backpropagation requires discontinuous processing where neurons must freeze their feedforward activity values while error signals flow backward, requiring the brain to pause information processing for hundreds of milliseconds before updating connections.The Brain’s Learning Algorithm Isn’t Backpropagation
- The backpropagation algorithm, while remarkably successful in machine learning, contradicts essential biological principles of brain function, making its exact implementation in neural tissue virtually impossible.The Brain’s Learning Algorithm Isn’t Backpropagation
- Neuroscientists are convinced the brain cannot implement backpropagation because synaptic signals only move in one direction, which motivates alternatives like Hinton's forward-forward algorithm that approximate backpropagation's good properties; but for pure engineering there is no reason not to use backpropagation, which remains the only algorithm.Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
- Using a trained network (e.g., ChatGPT inference) is purely a feedforward process where an input vector propagates layer by layer to an output, whereas training requires the back-and-forth of computing a loss on the output side and backpropagating it to update each layer's weights toward the input side.336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
- The main challenge with analog hardware is that converting analog outputs back to digital numbers for backpropagation is expensive and difficult, creating a barrier to scaling analog neural networks to large problems like ImageNetGeoff Hinton - Will Digital Intelligence Replace Biological Intelligence? | Vector's Remarkable 2024
- A language model's output is a probability distribution over its vocabulary for the next token, from which a token is sampled; during training the target distribution is one for the correct word and zero elsewhere, and backpropagation over hundreds of billions to a trillion parameters gradually nudges the predicted distribution toward the correct word across every sentence scraped from the internet.336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
- The most beautiful thing about deep learning is that it actually works—you start with small neural networks and backpropagation, which are inspired by the brain, and when you scale them up with large data, they produce capabilities that match brain functions, which is unbelievable given how simple the core ideas are.Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94