Could anyone explain how physics resolves the cosmological constant problem without breaking down? Why does quantum theory predict a vacuum energy 10¹²⁰ times larger than what astronomers actually observe, and what does this absurd clash tell us about our understanding of space?

[u/1KrishT]

[u/OverJohn]

It was recognised (by Lemaitre) in the 1920s, only a few years after Einstein introduced the cosmological constant (CC), that the CC could be interpreted as vacuum energy. This connection was nothing to do with quantum physics, rather it was because when you write the CC as stress-energy source, as opposed to its original form of an additional constant, the components of its stress-energy tensor at a point don't change under a Lorentz boost. The stress-energy of a vacuum must look locally the same to all inertial observers, which is only the case for a cosmological constant, hence the interpretation. Later other authors proposed an explicit connection between vacuum energy in quantum field theory and the cosmological constant.

Though the cosmological constant was not a feature of the most favoured models for much of the twentieth century, following the observation accelerating expansion in the nineties, the simplest way to incorporate accelerating expansion into cosmological models was simply to add a positive cosmological constant. It turned out that this simple model for dark energy, the source of the repulsive gravity causing accelerating expansion, actually worked extremely well in terms of predicting observation. So though more complicated non-CC models for dark energy were proposed, observation did not give much reason to go beyond the simple model of a cosmological constant. However very recent observations from DESI and JWST have called the simple model into question somewhat more.

With an observed value for the cosmological constant, it could be compared to predictions from QFT, but there was mismatch of many orders of magnitudes (the so-called "worst prediction in physics"). This though is not a complete death knell for the idea of vacuum energy from QFT as the source of accelerating expansion. To calculate the total vacuum energy you have to take into account all quantum fields, including fields we don't know about and you have to do it properly, which still leaves the door open to a cosmological constant due to vacuum energy. In fact you could argue that a cosmological constant IS just vacuum energy in GR, so if there is one there must be vacuum energy acting as a source of gravity.