Information Theory
The mathematical study of how information is quantified, encoded, and transmitted, founded by Claude Shannon.
As defined in cwt_tim-harford
Other definitions
A mathematical theory developed by Claude Shannon that quantifies information and establishes that language and other systems have statistical regularities that can be exploited for encoding, compression, and reconstruction
The AI Expert: "Super AI Will Be Unstoppable!" – What’s Coming NEXT | Stephen WolframMathematical field founded by Shannon that quantifies, stores, and communicates information, with applications from signal processing and cryptography to neurobiology.
#95 Claude Shannon
Where it comes up · 15
- #92 Ed Thorp and Claude Shannon4 claims
- #95 Claude Shannon3 claims
- Conversations with Tyler - Terence Tao3 claims
- 67 | Kate Jeffery on Entropy, Complexity, and Evolution3 claims
- The AI Expert: "Super AI Will Be Unstoppable!" – What’s Coming NEXT | Stephen Wolfram2 claims
- 79 | Sara Imari Walker on Information and the Origin of Life2 claims
- How LLMs Took Over The World2 claims
- The Four Sources of Alpha1 claim
and 7 more
Most important claims · 10 of 27Search all claims for “Information Theory” →
- Shannon's classified Project X work at Bell Labs on SIGSALY, a secure wireless system, showed that secrecy systems and noisy communication systems are nearly identical problems, and thinking about concealing messages with random noise motivated core insights of information theory.#92 Ed Thorp and Claude Shannon
- Mathematics is the soil of the STEM ecosystem: other sciences show what is possible, while mathematics proves limits, such as the maximum bits per second a wireless channel can carry, which cannot be discovered by engineering alone.Conversations with Tyler - Terence Tao
- Knowing the mathematical capacity limit tells you how close you already are to perfection; a wireless encoder at 98 percent of theoretical capacity justifies stopping further development rather than spending another $1 million.Conversations with Tyler - Terence Tao
- Data is raw presented material that is not valuable by itself, while information, in the information-theory sense, is what reduces uncertainty, so edge depends on turning data into information that can be acted on.The Four Sources of Alpha
- Shannon's information theory from the 1940s became essential to the modern economy only when cellphones and big data arrived decades later, showing that mathematics can take a long time to become useful.Conversations with Tyler - Terence Tao
- Claude Shannon's 1948 paper on information theory laid out the idea that language has statistical regularities (e.g., Q is usually followed by U, E appears more frequently than X) that could be exploited to understand encrypted or scrambled text, forming the foundation for statistical approaches to language generation.The AI Expert: "Super AI Will Be Unstoppable!" – What’s Coming NEXT | Stephen Wolfram
- Mathematics is developed more in the service of the world one lives in and the adjacent problems it solves than is typically recognized; information theory could have been defined purely much earlier but wasn't until error correction over real information channels at Bell Labs prompted it, and chaos theory required computers to even observe the phenomenon of sensitivity to initial conditions before the math could be discovered.Grant Sanderson (@3blue1brown) — Past, present, & future of mathematics
- Life should not be defined as chemical; rather, life is a process in which information organizes matter across many scales, because the physical substrate (molecules) need not be universal but the informational process organizing matter is what is common to all life.79 | Sara Imari Walker on Information and the Origin of Life
- Shannon proved that digital coding allows communication with reliable accuracy given any arbitrary amount of noise, and since a computer is fundamentally a communication device transmitting complex states billions of times per second, no computer would work without this; the trade-off is that a single misplaced bit can halt the whole machine, whereas analog systems are more robust against failure.153 - Possible Minds
- Claude Shannon's information theory framed language as a sequence of predictions where each word is chosen from a set of likely words given what came before; building on this, 1980s researchers trained tiny networks to predict the next word, and these networks spontaneously clustered similar words (verbs with verbs, nouns with nouns, words of similar meaning).How LLMs Took Over The World