inductive bias
The set of prior assumptions or predispositions a learner brings, allowing it to learn from limited data; what fills the gap between human and machine data requirements.
As defined in 343 | Tom Griffiths on The Laws of Thought
Where it comes up · 3
- 343 | Tom Griffiths on The Laws of Thought7 claims
- 230 | Raphaël Millière on How Artificial Intelligence Thinks4 claims
- Ilya Sutskever – We're moving from the age of scaling to the age of research1 claim
Most important claims · 10 of 12Search all claims for “inductive bias” →
- Large language models illustrate exactly how much data is needed to solve Chomsky's learning problem: if LLMs have learned something like human language, this proves Chomsky's point that learning language requires enormous data—a child cannot figure out language structure in five years, and it takes something like 5,000 to 50,000 years of data to do it well; the difference is made up by inductive bias from evolution and broader world experience.343 | Tom Griffiths on The Laws of Thought
- The big difference between human minds and current large language models is inductive bias: a human child learns language from about five years of exposure, whereas the data used to train an LLM is equivalent to between 5,000 and 50,000 years of continuous speech; the thing that makes up that orders-of-magnitude gap is the inductive bias—prior distributions broadly construed—that humans bring.343 | Tom Griffiths on The Laws of Thought
- The debate is often caricatured as either tabula-rasa empiricism or strong domain-specific nativism, but modern neural networks have inductive biases of a more general kind, suggesting a continuum in both the strength and domain-specificity of biases; and if evolution is viewed as a learning process, even 'innate' biological knowledge may be learned at the species scale.230 | Raphaël Millière on How Artificial Intelligence Thinks
- Giving AI models human-like inductive biases makes them less inclined to find solutions unlike human solutions, which is both a plus and a minus: a minus because humans are bad at many things and we may want AI to complement us by doing what people can't, but a plus because human-aligned systems learn from less data (easing energy concerns) and produce solutions that make more sense to humans.343 | Tom Griffiths on The Laws of Thought
- Over the recent history of AI, attempts to improve models by hard-coding human knowledge repeatedly underperform simply letting connectionist models learn from data bottom-up—a pattern Rich Sutton called the 'Bitter Lesson'—which holds across computer vision and language, where models induce grammar without any Chomskian innate universal grammar.230 | Raphaël Millière on How Artificial Intelligence Thinks
- The success of connectionist learning may nudge us toward a more empiricist view of human and animal learning, suggesting that things we thought unlearnable without innate priors—such as grammar—may in fact be learnable, putting pressure on claims like Chomsky's that universal grammar is innately encoded in DNA.230 | Raphaël Millière on How Artificial Intelligence Thinks
- Chomsky showed that language is more complex than behaviorists assumed, requiring internal structures like noun and verb phrases (grammars), which created the problem of how humans could learn such complex objects from limited data; Chomsky's answer was that they don't really learn it but acquire it via strong innate constraints that require only small amounts of data to fix the particular language being spoken.343 | Tom Griffiths on The Laws of Thought
- Even if an AI system reaches a point where it uses and produces language well, from the outside it may look similar to a human while internally being quite different—it may have found a very weird solution to language; analyses in Griffiths's lab suggest this is the case, and nudging models toward human-like inductive biases would yield solutions that make more sense to us.343 | Tom Griffiths on The Laws of Thought
- Current AI systems differ from human minds in two big ways: inductive bias (ability to learn from small data) and generalizability—an AI can be very good at one problem yet fail spectacularly on an adjacent one, a pattern called jagged intelligence; cognitive science is a good tool for understanding where those jagged boundaries lie and improving the systems.343 | Tom Griffiths on The Laws of Thought
- Neural networks are randomly initialized (weights start as random numbers), so in one sense they are a blank slate, but the architecture itself is not random—modern architectures embed substantial structure, or 'inductive biases,' that guide learning, so in another sense they are not a blank slate.230 | Raphaël Millière on How Artificial Intelligence Thinks