I think thermodynamics and statistical physics are really interesting. In Parallel Distributed Processing they discuss an extension of the entropy metaphor to learning and neural networks
https://mitpress.mit.edu/books/parallel-distributed-processi...
Has been a subject of interest for the last six decades, from http://compbio.biosci.uq.edu.au/mediawiki/upload/b/b3/Jaynes... to https://journals.aps.org/prl/pdf/10.1103/PhysRevLett.117.260... or https://arxiv.org/abs/1209.5500 to give some examples.
I'm pretty sure they're the same, with minor differences in notation and units, unless of course I misunderstand what you mean by "physical entropy". Entropy in physics only got a precise definition once physicists understood that the concept really comes from information theory.
could be totally wrong though
entropy, complexity, and evolution are words that come to mind
"The form of H will be recognized as that of entropy as defined in certain formulations of statistical mechanics where p_i is the probability of a system being in cell i of its phase space. H is then, for example, the H in Boltzmann’s famous H theorem."
where the general form was H = -K\sum p_i * log(p_i)
IIRC, this form is the only kind with the following two properties:
- chain rule -- H(X,Y) = H(X) + H(Y|X)
- maximized for the uniform distribution.
from wikipedia: "In mathematical statistics, the Kullback–Leibler divergence is a measure of how one probability distribution diverges from a second expected probability distribution"
Kullback-Leibler divergence is used a lot in the reinforcement learning setting
"In information theory, the cross entropy between two probability distributions p and q over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set"
Cross Entropy is used as a cost function in neural networks rather than least squares http://neuralnetworksanddeeplearning.com/chap3.html#the_cros...
I want to understand his work on random processes too, have just kind of scanned the book and have tried to upgrade my calculus