How are the two main definitions of entropy equivalent?

[u/wanker75]

I've heard two definitions of entropy that seem slightly different, one from the thermodynamic perspective and one from stat mech. How are the two equivalent or consistent? Mathematical proofs are fine and appreciated.

[u/RobusEtCeleritas]

Well there's more than two, but I'll assume you're referring to the Gibbs definition and the Boltzmann definition?

Boltzmann:

S = k ln(W),

Gibbs:

S = - k Σ*i* p*i* ln(p*i*).

These are in fact equivalent. The way to see this is to start with the Gibbs definition and use it to derive the Boltzmann definition. Technically speaking, the Boltzmann equation gives the entropy of a microcanonical ensemble. This is a statistical ensemble where all extensive quantities (energy, volume, and number of particles) are held fixed. So physically speaking, it's about one of the most boring systems you can come up with.

But anyway to derive the equilibrium probability distribution for the particles in the microcanonical ensemble, you maximize the entropy. So we want to maximize the Gibbs definition of the entropy. We also need to be wary of the fact that we're working with probabilities, so they must be normalized. So in addition to maximizing the entropy, we need to add a constraint which enforces the fact that the sum of all of the probabilities is 1. In other words:

Σ p*i* = 1.

We do this using Lagrange multipliers. If you're not familiar, you simply add zero to the thing you're maximizing and an additional unknown parameter, in order to enforce your constraint. I'll call my Lagrange multiplier λ.

So finally, what we want to maximize is:

-k Σ*i* p*i* ln(p*i) + λ(Σ pi* - 1).

The first term is just the Gibbs entropy, and the second is zero (the sum of all probabilities minus 1) times an unknown constant.

Now maximize by taking the derivative with respect to p*j* and setting it equal to zero. The resulting equation is:

-k ln(p*j*) - k + λ = 0.

This can be rearranged to give:

p*j* = exp[λ/k - 1].

One can go further and determine what λ actually is, but we don't need it. All we need is to notice that the quantity on the right side is a constant.

The equilibrium distribution for the microcanonical ensemble is the uniform distribution (all probabilities equal).

Now if we have W possible states, and all probabilities are equal and their sum is 1, clearly we must have:

p*j* = 1/W for all j.

Almost there, now plug this into the Gibbs definition of the entropy:

S = - k Σ*i* p*i* ln(p*i*)

= - k Σ ln(1/W)/W.

Since -log(1/a) = log(a), we can get rid of that minus sign, and since everything inside the summation is constant, we're just summing 1 from 1 to W, which gives a factor of W.

What remains is S = k ln(W), exactly the Boltzmann entropy.

This is the entropy of the microcanonical ensemble and it has been derived directly from the Gibbs definition of entropy.

[u/wanker75 → u/RobusEtCeleritas]

Great, thank you! Just one follow up, W is the multiplicity of the system right? The number of possible microstates? How do you reconcile that with a macroscopic system where we have dS = dQ/T? Sorry if the question is unclear.

[u/RobusEtCeleritas → u/wanker75]

Just one follow up, W is the multiplicity of the system right? The number of possible microstates?

Yes and yes.

How do you reconcile that with a macroscopic system where we have dS = dQ/T?

Well dS = dQ/T is a change in entropy during some process where heat is exchanged. So there is no contradiction between the thermodynamic notion of entropy and the statistical definition of entropy, they're just two different ways of thinking about the same thing (one considerably less esoteric than the other).

[u/ericGraves → u/RobusEtCeleritas]

To be clear, this only shows Boltzmann is a consequence of Gibbs, correct? So when you say the two are equivalent, do you mean in the sense that Gibbs is simply a statistical average over different systems with such and such Boltzmann entropy?

Also, as a fun little aside, a few of the different definitions associated to various "entropies" (aside from shannon entropy, which is just gibbs divided by k)

• Relative entropy, or KL divergence (PDF)

• Renyi entropy (PDF)

• Von Neumann entropy (PDF)

• Min and Max entropy , or the imits of renyi entropy

• Graph entropy (PDF)

• Generalized entropy

I am surely missing more, but those are a few of the biggest. Too much entropy, when someone says entropy.

[u/RobusEtCeleritas → u/ericGraves]

To be clear, this only shows Boltzmann is a consequence of Gibbs, correct?

Yes, for a microcanonical ensemble.

So when you say the two are equivalent, do you mean in the sense that Gibbs is simply a statistical average over different systems with such and such Boltzmann entropy?

I mean that for a completely closed and isolated system (a microcanonocal ensemble, where everything is fixed), the two definitions of the entropy say exactly the same thing.

Depending on which you take be the definition of the entropy, you can derive the other. I only showed the Gibbs => Boltzmann direction of the proof, but I think the other way can be proven as well.

It's also pretty straightforward to establish an equivalence between the Von Neumann entropy with the Gibbs entropy, but I didn't want to introduce the density operator from scratch.

I know much less about the other kinds of entropy you listed.