Hacker News Reader

Comment by cortesi | Hacker News Reader

Why, hello there Hacker News. Just a note that I've written two follow-on blog posts after this one, developing the idea a bit further.

First, using a color function that encodes local entropy to show how crypto keys and other high-entropy data can be picked straight out of a visualization:

http://corte.si/posts/visualisation/entropy/index.html

And next, using this in bulk on samples from a malware database, with what I think are beautiful and interesting results:

http://corte.si/posts/visualisation/malware/index.html

View on HN

I'm curious as to how it would look if you used some other measure of entropy besides local symbol frequency. Probably a general purpose compression algorithm should give you a good idea, like gzipping the blocks.

Hmm, I'm not sure if this could work, but perhaps if you use a stream based compression algorithm you could relatively precisely see how much compressed data it takes to represent up to a certain point in the file, with only a single pass (rather than having to compress a huge number of local windows). Of course this is probably going to weight earlier parts of the file heavier, simply because the compression won't be calibrated yet to efficiently encode. So you could also run it on a byte-reversed version of the file, and a "rotated" version of the file (i.e. file[n/2:n]+file[0:n/2]), and a rotated-byte reversed version, and combine all those metrics together in some way (maybe min(entropy1,entropy2,entropy3,entropy4)).

That way you could get an entropy measure which compensates for the sort of alphabet runs that fooled shannon entropy.

I'm working on something like this for the next evolution of the entropy visualizations. I've toyed with compression, but have had nicer results so far with other randomness estimators. I'll do a writeup once I have something to show.

For a continuous compression function your idea of running on a rotated or reversed version of the file and then taking a minimum is a cunning one! Right now, I'm still working with sliding windows, but I'll keep it in mind if I turn back to compression.

Very cool! I'm trying to visualize entropy based on Google Books' n-gram models of text.

I'm curious what other randomness estimators you'll be using... (or maybe I'll just have to wait for another blog post).

Beautiful work as usual. The entropy visualisations in particular stuck out for me, and I can imagine how useful it is to be able to see this kind of information at a glance while dissecting malware.

Cheers. :) I'm trying very, very hard to resist the urge to start writing a binary dissection tool based on this. I'm picturing a hex viewer with a space-filling curve navigation pane, with options to switch between different pixel layouts, and color maps. Then again, the last thing I need is another side-project.

Do most binaries have this high ratio of encrypted/obfuscated content? That is, would a checker which simply looked at high-entropy fraction of a binary be able to detect malware not in its database? Obviously this is trivial to defeat, but it might be the case that a broad defense such as this would force malware authors to let their code grow much bigger, which might in turn lead to other generic signatures. Would it be worthwhile?

Unfortunately perfectly legitimate compressed sections are both very entropic and very common, so just an entropy measurement won't be very useful for malware detection. I think there might be some promise in looking at patterns of entropy, though. What you can't see in that malware post is that very many pieces of malware show very similar layouts of entropy, even if the malware itself is very different. I weeded out similar images so that people could get an idea of the range of layouts, but the 50 visualizations on the blog are a "condensed" version of about 400 highly redundant images. Part of this is because different types of malware can still use the same "packer" - tools that pack a binary to obfuscate malware and make it hard to detect and reverse engineer. Part of it is just because there are certain techniques that are commonly used. All of this requires more study, but it's interesting.

This would be not only a wonderful toy, but a really fantastic tool. Of course I'm not trying to influence you in the slightest way... :)

Indeed; I highly recommend everyone read those posts. The original link is an interesting idea; the two follow-on posts are a surprisingly effective practical demonstration.

Reply on news.ycombinator.com