Hacker News Reader

Comment by prof-dr-ir | Hacker News Reader

If I would write a book with that title then I would get to the point a bit faster, probably as follows.

Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is:

S = - sum_i p_i log( p_i )

Mathematically we say that entropy is a real-valued function on the space of probability distributions. (Elementary exercises: show that S >= 0 and it is maximized on the uniform distribution.)

That is it. I think there is little need for all the mystery.

View on HN

So the only thing you need to know about entropy is that it's a real-valued number you can associate with a probability distribution? And that's it? I disagree. There are several numbers that can be associated with probability distribution, and entropy is an especially useful one, but to understand why entropy is useful, or why you'd use that function instead of a different one, you'd need to know a few more things than just what you've written here.

Exactly, saying that's all there is to know about entropy is like saying all you need to know about chess are the rules and all you need to know about programming is the syntax/semantics.

Knowing the plain definition or the rules is nothing but a superficial understanding of the subject. Knowing how to use the rules to actually do something meaningful, having a strategy, that's where meaningful knowledge lies.

In particular, the expectation (or variance) of a real-valued random variable can also be seen as "a real-valued number you can associate with a probability distribution".

Thus, GP's statement is basically: "entropy is like expectation, but different".

Of course that is not my statement. See all my other replies to identical misinterpretations of my comment.

The problem is that this doesn't get at many of the intuitive properties of entropy.

A different explanation (based on macro- and micro-states) makes it intuitively obvious why entropy is non-decreasing with time or, with a little more depth, what entropy has to do with temperature.

The above evidently only suffices as a definition, not as an entire course. My point was just that I don't think any other introduction beats this one, especially for a book with the given title.

In particular it has always been my starting point whenever I introduce (the entropy of) macro- and micro-states in my statistical physics course.

That doesn't strike me as a problem. Definitions are often highly abstract and counterintuitive, with much study required to understand at an intuitive level what motivates them. Rigour and intuition are often competing concerns, and I think definitions should favour the former. The definition of compactness in topology, or indeed just the definition of a topological space, are examples of this - at face value, they're bizarre. You have to muck around a fair bit to understand why they cut so brilliantly to the heart of the thing.

That definition is on page 18, I agree it could've been reached a bit faster but a lot of the preceding material is motivation, puzzles, and examples.

This definition isn't the end goal, the physics things are.

That covers one and a half of the twelve points he discusses.

Correct! And it took me just one paragraph, not the 18 pages of meandering (and I think confusing) text that it takes the author of the pdf to introduce the same idea.

You didn’t introduce any idea. You said it’s “just a number” and wrote down a formula without any explanation or justification.

I concede that it was much shorter though. Well done!

Haha you reminded me of that idea in software engineering that "it's easy to make an algorithm faster if you accept that at times it might output the wrong result; in fact you can make infinitely fast"

Thanks for defining it rigorously. I think people are getting offended on John Baez's behalf because his book obviously covers a lot more - like why does this particular number seem to be so useful in so many different contexts? How could you have motivated it a priori? Etcetera, although I suspect you know all this already.

But I think you're right that a clear focus on the maths is useful for dispelling misconceptions about entropy.

Misconceptions about entropy are misconceptions about physics. You can’t dispell them focusing on the maths and ignoring the physics entirely - especially if you just write an equation without any conceptual discussion, not even mathematical.

I didn't say to only focus on the mathematics. Obviously wherever you apply the concept (and it's applied to much more than physics) there will be other sources of confusion. But just knowing that entropy is a property of a distribution, not a state, already helps clarify your thinking.

For instance, you know that the question "what is the entropy of a broken egg?" is actually meaningless, because you haven't specified a distribution (or a set of micro/macro states in the stat mech formulation).

Ok, I don’t think we disagree. But knowing that entropy is a property of a distribution given by that equation is far from “being it” as a definition of the concept of entropy in physics.

Anyway, it seems that - like many others - I just misunderstood the “little need for all the mystery” remark.

Right, I see what you're saying. I agree that there is a lot of subtlety in the way entropy is actually used in practice.

> is far from “being it” as a definition of the concept of entropy in physics.

I simply do not understand why you say this. Entropy in physics is defined using exactly the same equation. The only thing I need to add is the choice of probability distribution (i.e. the choice of ensemble).

I really do not see a better "definition of the concept of entropy in physics".

(For quantum systems one can nitpick a bit about density matrices, but in my view that is merely a technicality on how to extend probability distributions to Hilbert spaces.)

I’d say that the concept of entropy “in physics” is about (even better: starts with) the choice of a probability distribution. Without that you have just a number associated with each probability distribution - distributions without any physical meaning so those numbers won’t have any physical meaning either.

But that’s fine, I accept that you may think that it’s just a little detail.

(Quantum mechanics has no mystery either.

ih/2pi dA/dt = AH - HA

That’s it. The only thing one needs to add is a choice of operators.)

Sarcasm aside, I really do not think you are making much sense.

Obviously one first introduces the relevant probability distributions (at least the micro-canonical ensemble). But once you have those, your comment still does not offer a better way to introduce entropy other than what I wrote. What did you have in mind?

In other words, how did you think I should change this part of my course?

Many students will want to know where the minus sign comes from. I like to write the formula instead as S = sum_i p_i log( 1 / p_i ), where (1 / p_i) is the "surprise" (i.e., expected number of trials before first success) associated with a given outcome (or symbol), and we average it over all outcomes (i.e., weight it by the probability of the outcome). We take the log of the "surprise" because entropy is an extensive quantity, so we want it to be additive.

Everyone who sees that formula can immediately see that it leads to principle of maximum entropy.

Just like everyone seeing Maxwell's equations can immediately see that you can derive the the speed of light classically.

Oh dear. The joy of explaining the little you know.

As of this moment there are six other top-level comments which each try to define entropy, and frankly they are all wrong, circular, or incomplete. Clearly the very definition of entropy is confusing, and the definition is what my comment provides.

I never said that all the other properties of entropy are now immediately visible. Instead I think it is the only universal starting point of any reasonable discussion or course on the subject.

And lastly I am frankly getting discouraged by all the dismissive responses. So this will be my last comment for the day, and I will leave you in the careful hands of, say, the six other people who are obviously so extremely knowledgeable about this topic. /s

The definition by itself without intuition of application is of little use

Don’t forget it’s the only measure of the arrow of time.

One could also say that it’s just a consequence of the passage of time (as in getting away from a boundary condition). The decay of radioactive atoms is also a measure of the arrow of time - of course we can say that’s the same thing.

CP violation may (or may not) be more relevant regarding the arrow of time.

Reply on news.ycombinator.com