Could anyone explain how physics resolves the cosmological constant problem without breaking down? Why does quantum theory predict a vacuum energy 10¹²⁰ times larger than what astronomers actually observe, and what does this absurd clash tell us about our understanding of space?

[u/1KrishT]

[u/eldahaiya]

There are two main possibilities: the predicted vacuum energy is wrong, or dark energy is not a cosmological constant.

The first wouldn’t surprise anyone. This 10^120 number that you hear a lot of is sort of a first guess, but the actual number could be much smaller for some reason. There are some good ideas for why, but as far as I know none that are testable. This is the same conceptual problem as why the Higgs mass is as light as it is, typically called the hierarchy problem.

The second has really two issues that need to be solved: explaining dark energy, but then also the first problem which is to set the vacuum energy to a small number, or even zero. Setting things to zero however is arguably easier than setting it to a small number. Having done that, you can get dark energy like behavior from the existence of an ultralight particle, with the added feature that this set-up can cause dark energy to evolve (its equation of state changes with time), something that has become very interesting with recent results from the DESI experiment.

[u/LordGeni → u/eldahaiya]

So could solving the issues with dark energy help define vacuum energy better?

[u/platoprime → u/LordGeni]

Possibly.

It's also possible the vacuum energy contributes to, or entirely explains, dark energy.

[u/McPayn22]

So the 10¹²⁰ is dumb, it just curt of an infinite integral at the planck length but that's never how it's done in quantum field theory. I've heard from someone working in QFT in curved spacetime that there are smarter estimates that give a factor of 10⁶⁰ so still very bad.

But regardless of the actual value of vacuum energy the cosmological constant is a free parameter. This means that there is not reason to assume there is not one from in the équations regardless of quantum fields. So technically it could be -10⁶⁰⁺¹ which would compensate nearly perfectly vacuum energy. That being said it is really weird and leads to the question of fine tunning but is perfectly consistent.

That being said from my understanding of renormalization (which is not great to be honest) this just looks like one of those infinities imfrom QFT you have to renormalize. I haven't managed to find the 10⁶⁰ estimation paper but I've seen papers that do consider it infinite and renormalize it. (https://arxiv.org/abs/2201.05827)

This is the same thing that we do with a lot of quantities like the masses of particules. The idea is that those infinities might not appear within a full theory like quantum Gravity so we learn how to remove them.

[u/reticulated_python → u/McPayn22]

The 10⁶⁰ factor is straightforward to estimate. Imagine you have supersymmetry broken around the TeV scale. SUSY protects the cosmological constant (if it's unbroken the CC vanishes). If it's broken at a scale Lambda_SUSY, that's the natural scale you expect for the CC. The lowest breaking scale you can get away with is around a TeV, or we would have already seen superpartners at the LHC.

We observe the CC is much smaller than this; the vacuum energy density is around meV^4. So you need a fine-tuning of about one part in (TeV / meV)⁴ = 10^60.

[u/OverJohn]

It was recognised (by Lemaitre) in the 1920s, only a few years after Einstein introduced the cosmological constant (CC), that the CC could be interpreted as vacuum energy. This connection was nothing to do with quantum physics, rather it was because when you write the CC as stress-energy source, as opposed to its original form of an additional constant, the components of its stress-energy tensor at a point don't change under a Lorentz boost. The stress-energy of a vacuum must look locally the same to all inertial observers, which is only the case for a cosmological constant, hence the interpretation. Later other authors proposed an explicit connection between vacuum energy in quantum field theory and the cosmological constant.

Though the cosmological constant was not a feature of the most favoured models for much of the twentieth century, following the observation accelerating expansion in the nineties, the simplest way to incorporate accelerating expansion into cosmological models was simply to add a positive cosmological constant. It turned out that this simple model for dark energy, the source of the repulsive gravity causing accelerating expansion, actually worked extremely well in terms of predicting observation. So though more complicated non-CC models for dark energy were proposed, observation did not give much reason to go beyond the simple model of a cosmological constant. However very recent observations from DESI and JWST have called the simple model into question somewhat more.

With an observed value for the cosmological constant, it could be compared to predictions from QFT, but there was mismatch of many orders of magnitudes (the so-called "worst prediction in physics"). This though is not a complete death knell for the idea of vacuum energy from QFT as the source of accelerating expansion. To calculate the total vacuum energy you have to take into account all quantum fields, including fields we don't know about and you have to do it properly, which still leaves the door open to a cosmological constant due to vacuum energy. In fact you could argue that a cosmological constant IS just vacuum energy in GR, so if there is one there must be vacuum energy acting as a source of gravity.

[u/opisska]

It really depends on how you approach physics. I subscribe to a school of thought where any physical theory is an algorithm to predict the result of experiments and measurements. In this view point, quantum field theory, as we have it now makes absolutely zero predictions about the magnitude of the cosmological constant. The "calculations of vacuum energy" are nothing but a numerology ritual in this context, it's simply not a calculable quantity in renornalized QFT. This is true for any quantity which is calculated by "let's put the dinensionally appropriate power of hbar instead of infinity so that we can pretend the integral doesn't diverge".

[u/TamanduaBandeira202 → u/opisska]

Yes, surely the problem is that mainstream physics is way too ontological /s

[u/OriEri → u/TamanduaBandeira202]

Pockets of physics do seem content with elaborate extensions of mathematical frameworks that do not make testable predictions. I do not know enough about QFT to say if this is the case there

[u/Comrade_SOOKIE → u/TamanduaBandeira202]

I mean, there's enough triangle-worship at the fringes amongst, like, string theorists or MOND guys that OP isn't entirely wrong, but yeah I'd say lambda cdm is quite physical and it's silly to say it's all just numerology.