I'd highly recommend Arieh Ben-Naim's books on entropy for anyone looking for more on this point.

TLDR he makes the argument that entropy is a measure of the probability of probability distributions.