359 | Solo: Theories of Dark Energy
What this covers
This is a solo episode that follows Sean Carroll's earlier treatment of the cosmological constant. It asks what it would take to build a believable theory of dynamical dark energy. He starts with the constraints: dark energy must be nearly uniform and nearly constant, or it would have left traces in the microwave background, lensing and large-scale structure. He then covers the coincidence problem, the odd fact that vacuum energy is only a few times the matter density today. He also explains why cosmic acceleration was welcome news for inflation, even though the energy scales involved are wildly different.
The middle of the episode works through quintessence. A slowly rolling scalar field needs a mass near 10⁻³³ eV, and coupling it to ordinary matter should produce fifth forces unless the couplings are suppressed by about 10⁻⁵. Carroll explains how an approximate shift symmetry offers technical naturalness, and how a pseudoscalar coupling to electromagnetism could rotate the polarization of distant light by about a degree. He then turns to phantom energy, where negative kinetic energy implies negative-mass particles and a catastrophically unstable vacuum. He also describes his own "VAMPs" proposal, in which one field plays both dark matter and dark energy.
The last stretch takes up modified gravity. It covers MOND and the numerical link between its acceleration scale and the Hubble parameter. It also covers f(R) gravity gravity) as a hidden scalar-tensor theory, and a 1/R term that can drive acceleration but cannot explain galaxy rotation curves. Carroll argues from effective field theory that anomalies appearing in weak-gravity regimes are hard to motivate. He also notes that a varying Newton's constant can fool an analysis into inferring w below −1.
Carroll argues that dynamical dark energy was a motivated but largely unsuccessful attempt to solve the cosmological constant and coincidence problems, so the cosmological constant remains the default explanation while the alternatives (quintessence, phantom energy, modified gravity) each face naturalness or stability problems.
- Dynamical dark energy does not improve the cosmological constant problem, since that puzzle persists even if vacuum energy is zero
- Scalar field models need extreme fine-tuning, and the phantom (w less than -1) option is unstable because negative kinetic energy yields negative-mass particles
- Current hints that w differs from -1 are not yet statistically significant, so the cosmological constant is still the best bet
The cosmological constant stays default dark energy, though dynamical alternatives add problems and deserve exploration.
- When dynamical dark energy was first proposed in the late 1990s and early 2000s, there was essentially no empirical reason to abandon the cosmological constant, which fit the observations well from the start, so the motivation was theoretical rather than data-driven.
“there was essentially zero empirical reason to do so, by which I mean there was no data that said, oh, the cosmological constant doesn't really fit very well. From the start, the cosmological constant fit quite well.”
- Observations from 1998 to about 2005 showed the universe at the critical density that makes it spatially flat, split roughly 70 percent into a cosmological constant-like component and 30 percent into matter, of which about 25 percent is dark matter and 5 percent ordinary matter.
“it's divided up, about 70% cosmological constant in the simplest model, and about 30% matter.”
- A constant vacuum energy is a single number that cannot change or surprise you, so it offers no extra data and no flexibility for explaining puzzles, which is why theorists hoped a dynamical field with time-dependent behavior could help.
“if you have just a number, the vacuum energy, there's really nothing you can do with it, right? It's just sitting there.”
- Inventing dynamical dark energy is expected to add more problems, not reduce them, since a new field brings its own naturalness and fine-tuning puzzles on top of the cosmological constant problem.
“In fact, what's gonna happen is we're gonna add more problems by inventing dynamical dark energy.”
- Making dark energy dynamical does not solve the cosmological constant problem, because that puzzle, why vacuum energy is so much smaller than its natural value, remains even if the acceleration is caused by something other than vacuum energy or the vacuum energy is zero.
“In fact, what's gonna happen is we're gonna add more problems by inventing dynamical dark energy.”
- Observational programs to probe dark energy are worthwhile even though, if dark energy is the cosmological constant, the probes would only refine one number without revealing underlying physics, because the same surveys generate broad cosmological data.
“maybe we learn something completely different. Maybe even though we're motivating doing these experiments by testing theories of dark energy, we're still doing cosmology.”
- Even with recent hints, Carroll judges that the smart money still bets that w equals minus one and dark energy is just the cosmological constant, given its strong theoretical virtues, while remaining open to scalar fields or modified gravity.
“the smart money is that w is minus one, that it is just the cosmological constant.”
- Dynamical dark energy was an interesting exploration that did not solve the big puzzles, so Carroll's default remains the cosmological constant while still treating exploration of dynamical models as healthy science.
“if you ask me right now... I still think that the vast amount of smart money is on the good old cosmological constant being the source of dark energy”
- Carroll considers the discovery that the universe is accelerating the most surprising and profound discovery in fundamental physics during his professional career, and believes it will remain one of the most important puzzles to understand.
“this discovery that the universe is accelerating is the single most surprising and profound discovery in fundamental physics in the time that I've been doing fundamental physics professionally.”
- The cosmological constant appears as a number in effective field theory with an expected natural size, but the observed value is smaller than that expectation by roughly 10 to the minus 122, a discrepancy that particle physicists did not anticipate and that likely points to underlying physics.
“And the actual number is smaller than the expectation by something like 10 to the minus 122.”
- Early supernova analyses, including the Garnavich et al. paper Carroll co-authored around 1999, found w equal to minus one fits the data well, with some room to let the dark energy density and w vary together.
“w equals minus one, the equation of state parameter corresponding to a real cosmological constant, fits great. No problems with it at all, perfect fit.”
- When cosmic acceleration was established in the late 1990s and early 2000s, physicists argued that the cosmological constant should not be assumed to be the cause, and that dynamical dark energy should be considered because the field had been surprised before.
“there was essentially zero empirical reason to do so, by which I mean there was no data that said, oh, the cosmological constant doesn't really fit very well. From the start, the cosmological constant fit quite well.”
- Carroll believes it is very hard to build a theoretically respectable model in which w is less than minus one, though he remains open to someone finding a clever way.
“I think that it's very, very hard to make a theoretically legitimate, respectable model in which something like that happens.”
- Carroll's recent cyclic universe paper is motivated by the assumption that a cosmological constant creates a cosmic horizon and finite-dimensional de Sitter space, and its implications need to be examined.
“it's 100% motivated by the idea that there's a cosmological constant and that gives rise to a horizon”
Dark energy should be nearly uniform, with its equation of state measured free of restrictive theoretical priors.
- In general relativity, energy is conserved in a universe with only matter, but not in one with radiation or dark energy, so a dark energy component with w less than minus one causes its total energy to increase as the universe expands, which may seem troubling.
“Energy is not conserved in general relativity if what you mean by energy is the energy on the right-hand side of Einstein's equation.”
- Dark energy must be nearly uniform across space and nearly constant in time, because if it clumped into galaxies its effects would have shown up in the microwave background, large-scale structure, lensing, and stellar orbits, and supernovae succeeded precisely because they respond to total cosmic energy rather than local clumps.
“if this new energy, this dark energy, were clumping into galaxies and clusters of galaxies, then you would see it in the microwave background, in large-scale structure, in gravitational lensing, in the orbits of stars around galaxies”
- The early approach to dark energy was phenomenological: rather than proposing a specific theory, researchers added the equation of state parameter as a free parameter, fitting its value along with the energy density, which gives room to play while admitting ignorance.
“I'm parameterizing the possibilities of my ignorance, right? I'm saying there's some theory out there, it's not quite the cosmological constant.”
- The constraints a data analysis produces depend on the theoretical possibilities one permits at the outset, so an analysis that excludes a class of models, like Newton's constant varying, can systematically misinfer parameters such as w.
“the constraints you get out depends on the theoretical possibilities that you allow for, that you put in when you start your analysis.”
- The equation of state parameter w, defined by pressure equal to w times energy density, is a measurable quantity: w equals minus one for a cosmological constant, and departures from that value determine whether dark energy density fades or grows with time.
“let P equal w times rho, where w is some number, a number that would be minus one if it were exactly the cosmological constant”
- The 1999 supernova paper analyzing dark energy cut off the equation of state at w equal to minus one, excluding values below it because energy conditions suggested they were unphysical, even though the data could have been examined without that prior.
“we cut off the values of w we were looking at at minus one. I might have said, but if you wanna look for w less than minus one, you should be allowed to do that because looking at the data is looking at the data.”
- If dark energy density is only nearly constant rather than exactly constant, the Hubble parameter is still nearly constant and the scale factor still curves upward, so the universe keeps accelerating, though sub-exponentially rather than exponentially.
“If you have an almost constant energy density, then you get an almost constant Hubble parameter and an almost... Well, you still get an accelerating universe. It's not quite exponential, okay? It's sub-exponential”
Light quintessence requires unnatural tuning, leaving dark energy's tiny mass unexplained.
- The Higgs boson mass is small relative to the Planck or grand unification scales, a puzzle called the hierarchy problem, but for particles like the electron and neutrino there are known symmetry reasons for their tiny masses, while the quintessence mass of 10 to the minus 33 electron volts is an even more unexplained new puzzle.
“For the Higgs boson, there's no good reason why. That's the hierarchy problem.”
- A light quintessence field coupling to ordinary matter should produce observable fifth forces and time-varying fundamental constants, and Carroll's estimate showed that to hide from experiment its couplings must be suppressed by about 10 to the minus five, which is unnatural.
“this new scalar field has to be... Have all of its couplings suppressed by something like a factor of 10 to the minus five.”
- Tracker models, in which the dark energy scalar field tracks the matter density over time, were proposed to explain the coincidence problem, but they did not become convincing and appear to be ruled out by data because the aspiration to explain more constrains models too severely.
“my impression is that that hope of getting tracker like behavior doesn't actually fit the data very well. So it's a simple ruled out by an ugly fact kind of situation.”
- For a slowly rolling quintessence field to produce today's dark energy, its mass must be roughly 10 to the minus 33 electron volts, which is the Hubble constant expressed in energy units and is vastly smaller than any known particle mass, so the tiny value demands explanation.
“the quintessence field has to have a mass of about 10 to the power minus 33 electron volts. So it's hugely tiny compared to any known particle physics scale.”
- Dark energy can be modeled as a slowly rolling scalar field whose potential energy is nearly constant and kinetic energy nearly zero, so it acts like a cosmological constant; theorists already had decades of practice with this idea from inflation.
“what you want is the potential energy is approximately constant, the kinetic energy is approximately zero. Then the potential energy in your scalar field is acting almost kind of like a cosmological constant.”
- The chameleon mechanism, proposed by Weltman and Khoury, gives light scalar fields an effective mass in dense environments near ordinary matter, which suppresses fifth forces in laboratories while allowing cosmological effects.
“give those scalar fields a temporary mass when they were in the vicinity of ordinary matter”
Dark matter explains galactic patterns better than modified gravity approaches like MOND.
- Dark matter predicts that the ratio of ordinary to dark matter is high in galactic centers and low in the outskirts because collisionless dark matter does not dissipate energy, while ordinary matter does, so the qualitative pattern seen in data follows naturally from dark matter but not from modified gravity.
“Dark matter doesn't do that, dark matter doesn't bump into itself and radiate away energy”
- Variable mass particles, or VAMPs, proposed with Greg Anderson, let a scalar field whose potential has no minimum be stabilized by the energy cost of giving mass to ambient dark matter, so the same field could act as both dark matter and dark energy, though inhomogeneities in the naive form cause instabilities.
“this was supposed to be a candidate for both dark matter and dark energy all at once.”
- Milgrom's MOND was motivated by a near-universal acceleration scale at which galaxy rotation curves transition from needing no dark matter to needing it, and that acceleration is numerically approximately equal to the present Hubble parameter, a coincidence that seemed to connect galactic dynamics to cosmic expansion.
“that acceleration due to gravity is the Hubble parameter today. Numerically, it's approximately the same numeric value as the Hubble constant.”
- Modified gravity theories like MOND that were meant to eliminate dark matter have been ruled out by cosmic microwave background anisotropy predictions, so they no longer work, though a few researchers keep working on them.
“the idea of MOND and other modifications of gravity made predictions for the microwave background anisotropies and they came out false, they have been ruled out by the data.”
Hidden modified-gravity degrees of freedom call for Bayesian, test-driven dark energy research.
- Measuring rotation of light polarization is difficult because telescopes are not built for it, so experimenters must trick instruments into the measurement, and new detectors are being designed specifically for this purpose.
“It turns out it's really, really hard to measure rotations of polarizations because that's usually not what the telescopes are built to do.”
- A function f(R) gravity theory can be rewritten as general relativity plus a scalar field, showing that tensor descriptions can hide additional degrees of freedom, so an accelerating solution can be visible in the equivalent scalar-tensor form.
“sometimes you can write a theory that you thought was just describing a certain tensor field like the metric, but it's actually describing other tensor fields as well.”
- Carroll's paper with Trodden, Turner, and Duvvuri was among the first to explicitly propose f(R) modified gravity as an explanation for cosmic acceleration, and it helped launch a large follow-up literature, though Carroll underrated its significance at the time.
“this can be used to explain the acceleration of the universe... the whole program of what is now called f of R gravity became super duper popular.”
- Scientists should treat dark energy questions as Bayesian credence assignments across scalar fields, modified gravity, and unknown alternatives, keeping space for new data to update them, and theorists should propose models and tests before data arrive.
“We're trying to be good Bayesians. We're trying to say there's a credence for it being a scalar field or modified gravity or various other kinds of things we haven't thought of yet.”
Exploratory models are valuable scientific tools for others even when they fail to solve the original problem.
- Theoretical ideas tell experimenters what to measure, as the acceleration discovery led people to measure the equation of state parameter, so physics requires an ongoing give and take between theory and experiment rather than waiting passively for data.
“if we hadn't discovered the acceleration of the universe, no one would be trying to measure the equation of state parameter of the dark energy.”
- Carroll admits he repeatedly underestimated the value of ideas he had worked out and set aside, because a new tool for others can matter more than the immediate application, and he tries to avoid this mistake as he gets older.
“what I didn't get right, and I constantly make this mistake, I'm trying not to make it as I get older now, what I realize is it's a new tool for people to play with.”
- After the f(R) gravity paper, many open questions remained, including whether such theories fit the data carefully, avoid experimental tests of gravity, which function of R is best, how the scalar couples to matter, and whether the chameleon mechanism is needed.
“Can you fit the data? ... Can you be avoiding all the various experimental tests of modified gravity”
- Carroll dismissed his 1/R modified gravity idea as uninteresting, but after others independently proposed it he co-wrote a paper, and f(R) gravity became popular, showing that a model can be a new tool for others even if it does not solve the original problem.
“what I realize is it's a new tool for people to play with.”
Phantom energy with w below minus one is hard to achieve and physically pathological.
- A time-dependent Newton's constant in scalar-tensor gravity can trick analyses into inferring w less than minus one, though hard to achieve while satisfying all observational limits.
“you could be tricked into getting a best-fit value of w that was a little bit less than minus one.”
- Later work by Arkani-Hamed and others on ghost condensation showed that negative mass particles could settle into a stable equilibrium hidden in a theory's details, but this does not produce a Big Rip.
“there was work by people like Nima Arkani-Hamed and others that talked about something called ghost condensation.”
- A scalar field with negative kinetic energy has particle excitations with negative mass, which allows processes like a proton decaying into a neutron while conserving energy, and summing over the resulting phase space makes such decays infinitely likely, so the vacuum would be catastrophically unstable.
“So basically, the paper that we wrote said, no, w cannot be less than minus one, because it would be catastrophically unstable.”
- Recent data hints suggesting w less than minus one are not believable in Carroll's view because the fitting analyses did too quick a job, and models with w above minus one can fit the data just as well.
“I think they did too quick a job in fitting the parameters, and it's pretty easy to find models that would fit the data even with w not being less than minus one.”
- Setting w less than minus one produces phantom energy whose density grows with time, and the scale factor then reaches infinity at a finite future time, a singularity called the Big Rip, which Caldwell and Kamionkowski proposed as a possible fate of the universe.
“what they showed is, it hits a singularity at a finite time in the future.”
- Phantom energy can be realized by putting a minus sign in front of the kinetic energy of a scalar field, which makes the field roll up its potential rather than down, and this is how Caldwell first constructed the phantom model.
“all you have to do is put a minus sign in front of the kinetic energy.”
Modified gravity could explain dark matter and acceleration but clashes with effective field theory expectations.
- If gravity is modified so that its effects appear only when gravitational fields are weak, a single modification could explain both the galactic rotation curves and cosmic acceleration, requiring only the present Hubble constant as an input, though this runs against effective field theory expectations.
“So if you can modify gravity to explain both of these things away, then you only need one number.”
- Adding a reciprocal term 1/R to Hilbert's action changes nothing for weak-field galactic dynamics because the Schwarzschild solution remains an exact solution, so the theory cannot explain galaxy rotation curves, though it can drive acceleration.
“the Schwarzschild solution... is 100% exactly also a solution for this new theory with one over R added to the action.”
- Effective field theory leads one to expect that new physics appears at high energies and that low-energy behavior remains normal, so theories where weak gravity or flat spacetime triggers anomalous behavior fly in the face of standard expectations.
“In effective field theories, you expect things to go crazy in the ultraviolet, at high energies. But you expect everything to just be normal and under control in the infrared.”
- Galaxy rotation curves measured by Vera Rubin and others do not decline with distance from the center as expected from visible matter; they flatten, implying more gravitational pull at the outskirts than ordinary matter provides, which cold dark matter explains by a halo with low density contrast.
“the rotation curves do not diminish to nothing. They more or less flatten out.”
- Both the dark matter phenomenon and cosmic acceleration appear when gravity is weak and curvature is small, which contradicts the effective field theory expectation that low-energy physics should be normal and under control, making modified gravity hard to motivate.
“both things, phenomena, that kick in when gravity becomes weak, when the curvature of spacetime becomes small”
- Dark matter and dark energy have not been directly detected and are inferred from spacetime behavior, so they could in principle be artifacts of modifying gravity on cosmological scales, though MOND is now ruled out by cosmic microwave background data.
“We have data from the cosmic microwave background and the idea of MOND and other modifications of gravity made predictions for the microwave background anisotropies and they came out false”
Pseudo-Nambu-Goldstone quintessence is testable through light polarization rotation.
- Caldwell, Dave, and Steinhardt named dynamical dark energy based on a scalar field quintessence, after the fifth element of ancient Greek physics, the heavenly element beyond earth, air, fire, and water.
“they dubbed the idea quintessence, the idea of dynamical dark energy.”
- An approximate shift symmetry in the quintessence field, making it a pseudo-Nambu-Goldstone boson, can technically protect both its tiny mass and its small couplings to other fields, because the symmetry is only slightly broken, which is what makes the smallness technically natural.
“This symmetry is protecting the mass. It's still a puzzle why it's so small. It's like really, really, really small, but it is technically natural”
- A slowly rolling pseudoscalar field can couple to electromagnetism only through the E dot B interaction, which survives the symmetry, and this rotates the polarization plane of light from distant sources by roughly one degree, a value below the then-current five-degree limit and therefore testable.
“it will rotate the plane of polarization of light from distant galaxies and the microwave background.”
- Recent analyses of cosmic microwave background data, notably by Komatsu, show a suggestion of polarization rotation at a tiny level, but it is not yet statistically significant, so it is only a suggestion that dedicated future data must confirm.
“it's not quite statistically significant yet, but it's getting there.”
Cosmic acceleration helped inflation and sharpens the coincidence problem.
- The coincidence problem asks why the vacuum energy density is only three to five times the matter density today, when matter density dilutes with expansion while vacuum energy stays constant, so earlier matter dominated and later vacuum energy will dominate; this near-equality at the present era is suspicious and suggests an anthropic explanation unless a new physical mechanism is found.
“By approximately the same, we mean three or four times or five times as much, something like that, but not 10 to the 100 times as much. This is called the coincidence problem”
- The discovery of cosmic acceleration was very good news for inflation for two reasons: inflation predicted a spatially flat universe at critical density, which the 1990s measurements of only about a third of critical density did not match until dark energy supplied the missing energy, and acceleration showed that vacuum energy is not forbidden from gravitating, undermining self-tuning ideas that would have killed inflation too.
“the discovery that the universe is accelerating was hugely good news for the inflationary universe scenario for two separate reasons.”
- Although inflation and dark energy share the same basic equations, the energy density needed for inflation is far higher than any particle physics scale probed experimentally, while the energy needed for present-day acceleration is extremely low, close to the average energy density of a mostly empty universe.
“The energy density that you need for inflation is hilariously high, much higher than any particle physics scale we've actually probed experimentally, whereas the energy you need for making the universe accelerate today is hilariously low.”
Career pressures and credit norms shape how physicists choose research and receive recognition.
- Carroll realized his productive but less broadly interesting papers were hurting his faculty job prospects, so he chose to work on the acceleration of the universe, which was interesting both to him and to the wider community, and the subsequent fifth-force-free symmetry idea helped him get faculty jobs.
“So I realized that I had to do something that was both interesting to me but also interesting to the rest of the world and make an impact there if I was going to get any faculty jobs.”
- Particle physics collaborations list authors alphabetically because individual contributions are impossible to track for large teams, whereas early supernova collaborations placed the lead author first and the rest alphabetically, and Carroll accepted being listed last as a non-member.
“So the strategy for our paper was Peter Garnavich came first, the whole rest of the High-Z Supernova Team came next in alphabetical order, and then I came last”
- Progress in fundamental physics is slow because a prediction made in 1999 about polarization rotation required decades to build telescopes capable of testing it, unlike fields that move quickly.
“it takes decades to make telescopes that are good enough to test that kind of prediction.”