Pure: A static analysis file format checker | Hacker News Reader

Pure: A static analysis file format checker

github.com

98 pointsemidoots14 comments

What a surprise to open up HN today and see Pure staring back at me!

Here's the HN thread from 2019 where rattray encouraged me just to go for it and open-source the original JavaScript version: https://news.ycombinator.com/item?id=20352439

That led to a contract with Microsoft, which then led to my work at Coil on TigerBeetle, a distributed database written in Zig: https://www.youtube.com/watch?v=rNmZZLant9o

Backstory: https://news.ycombinator.com/item?id=23760809

Thanks for the link to that comment of mine. Wow, that's a blast from the past!

Here's also a recent HN discussion about Pure, in the context of buffer bleeds and memory safe languages, with tptacek: https://news.ycombinator.com/item?id=31852820

What does "static analysis" mean in this context?

In program analysis it means that you're deducing properties of the program without executing it.

I think it's meant in the sense of "can detect the exploits without having to actually attempt decompressing anything"

Exactly, although, at present, Pure will also do dynamic analysis on the compressed stream once the metadata checks out, to verify that the compressed stream checks out with the metadata, i.e. that there's no semantic gap between the designated size and the actual stream size. This guards against one category of ZIP bomb.

Seems interesting to build something looking for things like zip file bombs, unexpected sparseness, overflow/underflow, and so on with C.

Is there something like this for formats other than ZIP like jpeg or pdf?

For tar: https://github.com/ferivoz/alquitran

I wanted to do this for those formats in particular, as well as Office files. Fastmail were going to sponsor some of this work to extend Pure, and I was chatting with the folks at Hey, but I couldn't get enough other sponsors to fund the work.

The idea is to use Pure to have a "Zero-Day Defense Mode" button for your email provider that you can push, if you want static analysis on 90% of the file formats coming and going, so that you can have stronger assurance on email attachments that you open. For example, I believe that Pure would have prevented last year's zero-click traversal against Apple Mail.

There are also so many ways to use file format anomalies to take out AV engines, so Pure can act as a first line of defense, before hostile data gets to them. For example, protecting them from the David Fifield zip bomb.

What I find most interesting with these checks, is that they're looking for explicit bleeds or explicit overflow bounds. So the signal to noise ratio is high.

Mostly, if people are interested, I would really love to move this all to Zig, to benefit from the checked arithmetic and spatial safety, since otherwise everything is all single-threaded run-to-completion.

Idk, but there's an interesting project, wuffs: a programming language specifically aimed at efficiently and safely parsing document structure. There is a PNG decoder [2].

1. https://github.com/google/wuffs

2. https://news.ycombinator.com/item?id=26714831

PDF is also a minefield, for how many ways there are to mess with the format: https://blog.didierstevens.com/2008/04/29/pdf-let-me-count-t...

This looks really interesting; can an end user make use of it directly? Or is it more for providers of things?

You can run it as a CLI directly on any ZIP file that you pass by path. I do want to port it to Zig though, to get rid of the NPM install bootstrapping, have a single binary, and also gain Zig's checked arithmetic and spatial safety.

Reply on news.ycombinator.com