358 | Solo: Vacuum Energy and the Cosmological Constant
What this covers
This solo episode, the first of two, traces the cosmological constant from its origin to the puzzles it now poses. Sean Carroll begins with Albert Einstein and the Einstein static universe, explaining why that balance fails: it must be tuned exactly, and real matter is lumpy, so gravity amplifies the overdense regions. He also disputes the myth that Einstein was philosophically committed to a static cosmos. Carroll says Einstein asked astronomers, took their answer, and adjusted the equations. Georges Lemaitre then shows the constant can be moved to the matter side as an energy density with negative pressure, which Carroll treats as the same physics rather than a rival theory. He also gives an intuitive route to acceleration through a constant Hubble parameter and a fixed doubling time.
The observational story follows. Carroll explains why Type Ia supernovae are standardizable, describes Saul Perlmutter's push for systematic surveys against astronomers' skepticism, and notes that by 1997 other data already disfavored a flat, matter-only universe. Matter at about 0.3 of critical density plus inflation's flatness implied roughly 0.7 in vacuum energy.
The rest turns to theory. Carroll covers zero-point energy, Zeldovich's 1960s insight that gravity feels vacuum energy, and the roughly 10^122 mismatch from effective field theory. He reviews failed fixes: supersymmetry, which is broken in nature, and his own self-tuning proposal, which Steven Weinberg showed was really tuned. The contested closing point is Weinberg's 1980s anthropic prediction of a small nonzero constant, which Carroll says proves nothing about the principle itself.
Carroll argues the cosmological constant, equivalent to vacuum energy, is the leading explanation for cosmic acceleration, but its observed value is wildly smaller than quantum field theory predicts, and the coincidence problem remains unresolved, so the anthropic principle is currently the least bad option.
- Lemaître showed the cosmological constant and vacuum energy are mathematically identical, so the open question is its measured value rather than its interpretation
- Effective field theory predicts a vacuum energy about 10^122 times larger than observed, and no proposed mechanism (supersymmetry, wormholes, self-tuning) has solved this
- Matter dilutes while vacuum energy stays fixed, so the near-equality of their densities today is the unexplained coincidence; anthropic reasoning is the best available response
Vacuum energy's natural gravitational size exceeds the observed value by about 10^122, a severe naturalness problem.
- Using effective field theory with an ultraviolet cutoff at the Planck scale, the predicted vacuum energy exceeds the observational limit by a factor of about 10^122, because the cosmological constant is a highly relevant parameter that dominates in the infrared.
“the answer is a famous factor of 10 to the 122 times bigger”
- In the 1960s Yakov Zeldovich pointed out that the vacuum energy in quantum field theory is what enters gravity as the cosmological constant; particle physics is insensitive to the overall vacuum energy, but gravity cares about total energy, creating a large problem because the natural scale is far above observations.
“Gravity cares about the total amount of energy in the universe, and Zeldovich appreciated that.”
- In 1990 most theorists assumed the cosmological constant was probably exactly zero, reasoning that more possible theories set it exactly zero than set it to a tiny nonzero value, though the anthropic principle was the main exception.
“more of them make it exactly zero than make it 10 to the minus 122 times its natural value”
- If vacuum energy were near its natural particle-physics value, the universe would have blown apart within a fraction of a second, so the discrepancy is so large that it is obvious without precise cosmological measurement, and the sign of the constant is not predicted either.
“The discrepancy is just so big that it's absolutely, overwhelmingly obvious.”
- Matter density falls as the universe expands while vacuum energy density stays fixed, so vacuum energy would be invisible early and dominant late; that we live in the brief era when the two are comparable is the coincidence problem, which Carroll found very convincing.
“And we just happen coincidentally to live in the one time in the history of the universe when these two numbers are comparable.”
- Although a constant can be measured and accepted as is, many theoretical physicists believe the huge gap between expected and observed vacuum energy is a clue that a better theory is needed, and some suspect effective field theory breaks down when gravity is involved.
“maybe you're missing an important clue that nature is trying to give you to develop a better theory”
- Carroll insists the cosmological constant and vacuum energy are not two distinct theories but precisely identical, and anyone who claims otherwise is wrong, which he stresses because it sets up the measurement problem.
“They're literally exactly the same thing.”
- For vacuum energy the pressure equals minus the energy density, so it acts like a tension, similar to a stretched rubber band, rather than a push.
“for the vacuum energy, Lemaitre pointed out that the pressure was negative”
- Theory predicts at most the absolute magnitude of the cosmological constant, not its sign, so it is more a guesstimate than a prediction.
“You're not making a prediction for its sign. You're making a prediction for its absolute magnitude, if anything.”
- If vacuum energy were anywhere near its natural particle-physics value, the universe's expansion would have torn it apart in a tiny fraction of a second after the Big Bang, so atoms, molecules, planets, and stars could never have formed.
“The expansion rate of the universe driven by such a vacuum energy would be so big that you couldn't make an atom”
Accelerating expansion is well established, fitting roughly 0.7 vacuum energy within Lambda-CDM.
- In fundamental physics there have been few helpful, experimental, surprising discoveries in past several decades, since the 1970s, apart from things like neutrino masses; the Higgs boson and gravitational waves were expected, so progress is hard because experiments drive great ideas.
“there haven't been a lot of helpful, experimental, surprising discoveries in the past several decades”
- Saying vacuum energy's negative pressure pushes galaxies apart works only through the rule that acceleration depends on energy density plus three times pressure (rho + 3p), which is mathematically correct but gives no intuition, so Carroll prefers a longer chain of reasoning.
“where do I get any intuition for the claim that rho plus 3p is appearing on the right-hand side of my equation?”
- In the Friedmann equation the squared Hubble rate is proportional to energy density; if energy density is constant, as for vacuum energy, the Hubble rate is constant, so the universe's doubling time is fixed and expansion is exponential, which is an intuitive route to acceleration.
“So a constant energy density leads to a constant expansion rate, which shows up as an exponential expansion to the universe”
- In 1998 two competing astrophysics groups using Type Ia supernovae to measure the expansion history found the universe is accelerating rather than decelerating, contrary to expectations, which is the most surprising discovery in fundamental physics in Carroll's career.
“the universe is actually accelerating, not decelerating, in its expansion rate”
- By 1997 observational cosmology had accumulated enough independent data to rule out the theorists' favored flat, matter-only universe with Omega equal to one, even though the alternative explanation was not yet known.
“there's enough data here already in 1997 to say that the theorists' favorite model of Omega equals one”
- Type Ia supernovae are approximately standard candles because they are white dwarfs reaching the Chandrasekhar limit; their remaining brightness differences are corrected through the Phillips relation linking light curve timescale to absolute brightness, making them standardizable.
“they're standardizable candles even though they're not standard”
- Astronomers measured matter density near 0.3 of critical density, and combined with inflation's flat-universe prediction this suggested 0.7 of the critical density is vacuum energy, which was allowed by 1990 data but seemed contrived.
“0.3 of it is matter and 0.7 of it is the cosmological constant”
- Around 1990 gravitational lensing statistics was thought to be the best way to constrain the cosmological constant, but supernova distances and CMB anisotropies superseded it within 10 to 15 years.
“this was thought to be the best way to constrain the cosmological constant”
- Early-2000s CMB measurements from ground telescopes and WMAP confirmed the supernova picture: about 0.7 of critical density is vacuum energy and 0.3 is matter, the Lambda-CDM model that still fits about twenty years later.
“the CMB confirmed the picture of the universe that had been put together by the supernova data”
- Discoveries that contradict everything physicists think they know are rightly met with skepticism, whereas discoveries that make existing pieces snap together are believed quickly, which is why the supernova acceleration result was accepted rapidly.
“Whereas if you claim a dramatic discovery that makes everything fit together that we think we know, people are much more likely to believe it right away.”
- Saul Perlmutter, a particle physicist, argued that supernova cosmology should use a dedicated systematic survey rather than waiting for chance discoveries, and his Supernova Cosmology Project did this with Department of Energy money, while astronomers were skeptical.
“Particle physicists think big. They don't want to take pictures of galaxies and hope something interesting happens.”
- Perlmutter's group was right that a systematic search could work, while the High-Z team was right that careful reduction was needed, and both approaches together produced the 1998 result.
“Saul was absolutely right in saying that it could be done”
- The High-Z supernova team's first paper, titled as a measurement of deceleration, ironically reported acceleration, and they doubted their own result enough to check for programming or data-entry mistakes before publishing.
“The irony being that what they actually ended up measuring was the acceleration of the universe.”
- Type Ia supernovae arise when a white dwarf accretes mass until it reaches the universal Chandrasekhar limit, so the explosions are roughly uniform in brightness, unlike core-collapse supernovae from red giants, which vary with their progenitors.
“they're standardizable candles even though they're not standard”
- The 1992 COBE satellite detected temperature anisotropies in the cosmic microwave background, which had to exist because galaxies must have grown from tiny early ripples, confirming a prediction long worked out by theorists.
“the COBE satellite did find them”
- The Hubble expansion rate cannot be thought of as a velocity because velocities differ between galaxies; it is better understood as a rate that determines how quickly the universe doubles in size, roughly the Hubble time.
“the Hubble time, which is one over the Hubble constant, is almost exactly that, the doubling time for the universe”
Anthropic reasoning is currently the best answer to the constant problem, though speculative and unproven.
- Carroll argues that the anthropic principle is currently the best available solution to the cosmological constant and coincidence problems, even though it is messy, involves speculative physics, and is not known to be right.
“the best solution that we have on the market right now is the anthropic principle. And that's not a good solution.”
- Sidney Coleman argued that Euclidean quantum gravity wormholes contribute to the parameters of particle physics, including vacuum energy, and calculated that their contribution cancels other contributions; but the result was fragile and did not survive scrutiny, leading Coleman to call it a house doubly built on sand.
“this is a house doubly built on sand”
- Tom Banks and Willy Fischler argue that if the dimension of Hilbert space is finite, the cosmological constant is related to that dimension and so is not an ordinary effective field theory parameter, which would completely change the problem.
“the dimensionality of Hilbert space is related to the cosmological constant”
- Cosmological phase transitions change the vacuum energy in the way that changes the average energy between water and ice, which is why ice floats, so spacetime and quantum fields can undergo energy shifts at phase transitions.
“When there's a phase transition, the energy changes.”
Proposed fixes such as supersymmetry and self-tuning have failed or remain unproven.
- Self-tuning brane models could cancel the observed vacuum energy, but when Carroll and Mersini replaced the energy density with rho plus p in the Friedmann equation, radiation's positive pressure contradicted nucleosynthesis, and Weinberg's no-go reasoning implied the model was fine-tuned after all.
“radiation has pressure. It has positive pressure. In the case of radiation, the pressure is one-third of the energy density.”
- Steven Weinberg proved that inventing new fields cannot solve the cosmological constant problem without fine-tuning in ordinary quantum field theory; Carroll's self-tuning proposal was, as Weinberg pointed out, not self-tuning but tuned.
“That's not self-tuning. You tuned it.”
- Supersymmetry relates particles of different spins and therefore constrains vacuum energy, so perfect supersymmetry could make it exactly zero; but supersymmetry is broken in nature (no selectron found), and breaking it generally produces a vacuum energy incompatible with data, so the approach seemed to fail.
“the vacuum energy would be exactly zero. You could actually solve the puzzle”
- Steven Weinberg showed that in a multiverse with varying cosmological constants, life-friendly regions would have small constants, and in the 1980s he predicted the constant would be nonzero and discoverable, which turned out roughly correct, though this does not prove the anthropic principle.
“the natural expectation... is that there is a non-zero value for the cosmological constant that we will eventually discover”
Acceleration may stem from dynamical dark energy rather than a constant, so alternatives remain open.
- If the acceleration is produced by a true cosmological constant, it will never go away, so the universe will keep accelerating, getting emptier, colder and more desolate forever, with a vacuum energy density of about 10^-8 ergs per cubic centimeter.
“There's no reason for that not to continue forever. The cosmological constant doesn't go away.”
- Although the cosmological constant fits the data very well, Carroll argues that because acceleration was a surprise, physicists should remain open to the possibility that the acceleration is caused by something other than a constant, such as dynamical dark energy, and that the constant could still be zero.
“maybe that thing is not the cosmological constant”
- Dark energy is the catch-all term for whatever causes acceleration, including the cosmological constant as one form, and other forms may change slowly with time, which opens many possibilities and means the constant is not the only option.
“The cosmological constant itself is a form of dark energy, but there are other forms.”
- The Hubble tension, a discrepancy in measured expansion rates, is not explained by current theory, so it is not yet clear whether it will go away, but such open tensions are what drive progress in science.
“The Hubble tension is something where we don't have a good theoretical model.”
Einstein's cosmological constant, equivalent to vacuum energy, was introduced for a static universe that failed without a Big Bang.
- Einstein, trying to build a finite, uniformly filled spherical universe with no expansion, added a positive cosmological constant term to his field equations so that spatial curvature and matter balanced and the universe stayed static, after finding his original equations implied it would expand or collapse.
“he added a new term to the equation, and he didn't do it just willy-nilly”
- The Einstein static universe requires matter density and cosmological constant to be balanced exactly, so any slight mismatch makes it eventually expand or contract; and real inhomogeneities grow over time through gravitational instability, so a perfectly uniform static universe cannot describe reality.
“Overdense regions, regions in which there's a little bit more matter than average, will pull more matter onto them.”
- Hubble's redshift-distance data showed the universe is expanding, making the static assumption obsolete; the story that Einstein called the cosmological constant his greatest blunder comes from George Gamow and is probably true, but Carroll thinks dropping the term because it is unneeded is the wrong scientific move.
“the scientific thing to do is not to say, 'I don't need it, I'm going to set it aside,' but to say, 'Is it there?'”
- Although Einstein was influenced by Ernst Mach and believed Mach's principle linked matter to inertial frames, the answers Einstein reached are disconnected from Mach's ideas, and people keep trying to force the connection.
“the answers that Einstein came up with are completely disconnected from any of Mach's ideas”
- Lemaître showed the cosmological constant can be moved from the geometry side of Einstein's equation to the matter side, where it is an energy density of empty space with negative pressure, and this is exactly the same physics as the original term, not a different theory.
“there literally isn't any difference between those two ideas”
- In Newtonian gravity, a universe with uniform matter density has no uniquely defined gravitational potential because a constant can be added, which made the whole-universe case puzzling and motivated Einstein's finite-space approach.
“it is undefined what potential you should have. There are multiple solutions that work equally well.”
- Real matter is lumpy, and gravity amplifies overdense regions, so a static uniform universe cannot persist; Einstein's model also lacked any Big Bang to explain why structure had not already formed.
“Denser regions grow more dense, less dense regions grow less dense, and the universe becomes lumpier and lumpier over time”
- Einstein preferred a finite spherical space on philosophical grounds, while Carroll says he has no strong prior and would let data or better theory decide whether space is finite.
“I'll let the data or a better theory decide that.”
- Einstein's equation sets the spacetime curvature side, built from the metric and Riemann tensors, proportional to the stress-energy tensor multiplied by 8 pi G, so matter and energy determine the geometry.
“So the right-hand side says that there's matter and stuff. The left-hand side says there's spacetime curvature. And the equation sets them proportional to each other.”
Zero-point energy is arbitrary in particle physics and matters only to gravity.
- The harmonic oscillator's zero-point energy of one-half h-bar omega is the difference between quantum and classical ground-state energies, not an intrinsic energy; since adding a constant to the energy has no physical effect in particle physics, the zero point can be set to any value, so it does not by itself show vacuum energy.
“the number one-half h-bar omega isn't even anything intrinsic to nature”
- Each mode of a free quantum field behaves as a harmonic oscillator with its own zero-point energy, and summing over all wavelengths gives infinite vacuum energy in naive quantum field theory, but renormalization sets it to an arbitrary value, so the lesson is that the vacuum energy is arbitrary and must be measured.
“The energy of empty space, the vacuum energy, is infinity, if you're very, very naive about quantum field theory.”
- Within the standard model of particle physics, only differences in energy matter, so the absolute vacuum energy can be ignored; it becomes relevant only when gravity is included.
“for the rest of particle physics... you can ignore gravity and you don't need to worry about the cosmological constant”
- Wheeler and Feynman's absorber theory, which treated electrons as particles and positrons as electrons moving backward in time, was partly motivated by worry about the infinite vacuum energy from quantum field theory; it failed but led to Feynman diagrams.
“they were just kind of worried about the fact that when you quantize field theory, you get an infinite contribution to the energy”
- Phil Anderson proposed a version of the Higgs mechanism, a scalar field with nonzero value in empty space, but did not pursue it, likely because the field would carry enormous energy density, which bothered him; he received no credit for the Higgs mechanism.
“he knew that there would be a tremendous energy density associated with that field”
Physicists repeatedly invoked and dropped Lambda, so the Einstein blunder story is a myth.
- Ever since Einstein proposed the cosmological constant, physicists repeatedly invoked it to solve observational puzzles and then dropped it when those puzzles went away, so its reputation cycled.
“there was a sort of a cycle of phases of people suggesting that the cosmological constant could help”
- The story that Einstein called the cosmological constant his greatest blunder was apparently told by George Gamow, who liked telling stories, though Carroll thinks it might be true.
“That was a story that was apparently told by George Gamow, who liked telling stories”
- The myth that Einstein was philosophically devoted to a static universe is false; he asked astronomers whether the universe was expanding, accepted their report that it looked static, and then changed the equations to fit that data.
“I'm going to listen to the data”
- Steven Weinberg wrote an influential late-1980s review of proposed solutions to the cosmological constant problem, and Carroll's own early review with Press and Turner was an astronomy-side complement to it.
“He had written an extremely influential review article in the late '80s”
- Cosmology was not considered a respectable part of physics from the 1920s through the 1960s because the ratio of theory to data was large, so the best physicists moved to quantum mechanics and particle physics.
“cosmology was not at that time, 1920s, even in '30s, '40s, '50s, maybe '60s, not the most respectable part of physics”
- Carroll says his perspective on dark energy and acceleration is not unbiased because he worked extensively on it, so he is presenting what was happening around him rather than a neutral history.
“So I'm not gonna try to give you a completely unbiased view of anything.”