Static analysis of an unknown compression format

Something I wrote a while back, using Python and PIL.

    import sys
    import math
    from PIL import Image

    def getSize(len, width):
        h = int(math.ceil(float(len) / width))
        return (width, h)
        
    def getImage(file, width=256):
        d = file.read()
        s = getSize(len(d), width)
        im = Image.new('L', s)
        im.putdata(d)
        return im
        
    def main():
        if len(sys.argv) < 2:
            print 'Usage: file2img.py <file> <output> [width]'
            return
            
        with open(sys.argv[1], 'rb') as f:
            if len(sys.argv) > 3:
                width = int(sys.argv[3])
                img = getImage(f, width)
            else:
                img = getImage(f)
            img.save(sys.argv[2])
        
    if __name__ == "__main__":
        main()

Here's a visualization of user32.dll on Windows. Notice how you can actually make out the images that are embedded in the resources!

http://i.imgur.com/7yEHR.png

You might find this useful: http://corte.si/posts/visualisation/binvis/index.html

and related: http://corte.si/posts/visualisation/entropy/index.html (visualizing entropy in binary files)

I used his software on an unknown format from the 1988 game Circuit's Edge to find out if/where the file was encrypted or compressed. It works well and was able to give me a (very) general idea of where in the file I needed to look further.

Edit: Nope, I remembered incorrectly. I used it on the game's .exe to tell me if I was dealing with a compressed executable. It turned out that it was compressed and so I went on a search for an ancient unpacker.

See before and after:

edge.exe packed: http://imgur.com/PlHGl

edge.exe unpacked: http://imgur.com/jeFLb

In my favorite static analysis toolset (Radare, http://radare.org/), there's a nice feature similar to this. It "zooms" out on a big file by summarizing portions of it into chunks and displaying colored ascii characters in a chart to represent the block.

With the entropy and ASCII printable filters setup, it makes repeating patterns and TEXT sections pretty clear.

It is very simple to create the binary versions of http://en.wikipedia.org/wiki/Netpbm_format from the command line, if you have that binary chunk in a file. Something like:

  echo P4 > file
  echo "640 480" >> file
  echo 255 >> file
  cat binaryBlob >> file

When doing simple graphics work at university, I would name files that contained nothing but pixel data foo.raw and IIRC some image editors would offer the options you mention when opening the files, then display them fine. Maybe that was before .raw got associated with cameras (and presumably more complex).

Both GIMP and Photoshop can read any binary file as a "raw" file format, where you can specify the bit-depth (e.g. greyscale, RGB, RGBA, etc) and column width.