The easy way: compress it with whatever you want (xz? gzip?) and compare the sizes. If it doesn't get significantly smaller, it's probably encrypted or already compressed.

binwalk -E

Will show a graph of the file offset vs shannon entropy.

read "A Mathematical Theory Of Communication" by Shannon himself, it's the only source you will ever need, it's available for free