Shannon entropy
A number assigned to a probability distribution measuring the amount of surprise, uncertainty, or information it contains; zero for a certain outcome and maximal for a uniform distribution.
As defined in 174 | Tai-Danae Bradley on Algebra, Topology, Language, and Entropy
Where it comes up · 5
- 174 | Tai-Danae Bradley on Algebra, Topology, Language, and Entropy2 claims
- 104 | David Rosen and Scott Miles on the Neuroscience of Music and Creativity1 claim
- 198 | Nick Lane on Powering Biology1 claim
- 360 | Marc Berman on the Science of Touching Grass1 claim
- AMA | June 20221 claim
Most important claims · 6 of 6Search all claims for “Shannon entropy” →
- Shannon entropy satisfies an equation strongly resembling the Leibniz product rule from calculus, and Bradley's paper makes this precise: entropy is a derivation, and conversely any function in the appropriate context satisfying the Leibniz rule is essentially entropy up to a constant multiple.174 | Tai-Danae Bradley on Algebra, Topology, Language, and Entropy
- Harmonic surprise can be quantified by treating each chord as an event, computing its frequency across a corpus (e.g., 30,000 chords from 600 major-key Billboard songs, all normalized to one key and truncated at sevenths), then taking the negative log base two of its probability to yield a value in bits—where common chords like C major score around one or two bits and a rare chord like an F-sharp minor ninth scores around seven bits.104 | David Rosen and Scott Miles on the Neuroscience of Music and Creativity
- Shannon entropy is a number assigned to a probability distribution that measures the amount of surprise or uncertainty it contains: a certain event (probability 1) has zero entropy because its occurrence conveys no information, while more spread-out distributions have higher entropy.174 | Tai-Danae Bradley on Algebra, Topology, Language, and Entropy
- Thermodynamic (Boltzmann) entropy increases while Shannon entropy stays constant because we don't track the full microstate; in quantum mechanics, branching allows the universe's initial quantum state to be genuinely simple (low information) yet split into many complicated branches whose combination remains simple — like cutting a simple circular plate into many complex jigsaw pieces that reassemble into something simple — which is impossible in deterministic classical mechanics where specifying initial and final states is equally hard.AMA | June 2022
- Life is driven by entropy increasing in the universe—self-organization like phase separation or protein folding releases heat and raises overall entropy—so rather than lamenting the second law of thermodynamics as everything running down, we should recognize that we exist only because entropy is increasing.198 | Nick Lane on Powering Biology
- Carroll conjectures, using Shannon's information theory, that natural scenes carry low-entropy information because the viewer is not surprised by a tree and can expect what it looks like, whereas urban scenes carry higher-entropy information with unexpected signs and details that demand more processing.360 | Marc Berman on the Science of Touching Grass