Hacker News Reader

Correct anything thats wrong here. Cross entropy is the comparison of two distributions right? Is the objectivity sussed out in relation to the overlap cross section. And is the subjectivity sussed out not on average but deviations on average? Just trying to understand it in my framework which might be wholly off the mark.

View on HN

Cross entropy lets you compare two probability distributions. One way you can apply it is to let the distribution p represent "reality" (from which you can draw many samples, but whose numerical value you might not know) and to let q represent "beliefs" (whose numerical value is given by a model). Then by finding q to minimize cross-entropy H[p, q] you can move q closer to reality.

You can apply it other ways. There are lots of interpretations and uses for these concepts. Here's a cool blog post if you want to find out more: https://blog.alexalemi.com/kl-is-all-you-need.html

I'm not sure what you mean by objectivity and subjectivity in this case.

With the example of beliefs, you can think of cross entropy as the negative expected value of the log probability you assigned to an outcome, weighted by the true probability of each outcome. If you assign larger log probabilities to more likely outcomes, the cross entropy will be lower.

Reply on news.ycombinator.com