Hacker News Reader

Comment by petertodd | Hacker News Reader

That's basically how I designed the magic bytes for the OpenTimestamps proof files:

    $ hexdump -C foo.ots 
    00000000  00 4f 70 65 6e 54 69 6d  65 73 74 61 6d 70 73 00  |.OpenTimestamps.|
    00000010  00 50 72 6f 6f 66 00 bf  89 e2 e8 84 e8 92 94 01  |.Proof..........|

0) Magic is at the beginning of the file.

1) Starts with a null-byte to make it clear this is binary, not text.

2) Includes a human-readable part to make it easy to figure out what the file is in hex dumps.

3) 8 bytes of randomly chosen bytes, all of which greater than 0x7F to ensure they're not ASCII.

3) Finally, a one-byte major version number.

4) Total length (including major version) is 32 bytes to fit nicely in a hex dump.

View on HN

These days I generally advise that you interpret the version number as odd and even bits: odd means it's compatible with readers, even means it isn't.

That sounds interesting. Can you say a little more about how this works?

It's a trick I stole from ext2, and simplified. In that filesystem there are three bitsets: one for reading, one for writing, one for fsck. If you don't understand a bit you can't do that action.

For most protocols there's only reading and writing, so you can use odd bits to mean "backwards compatible features, you can read even if you don't understand" and even for "stop, we broke compat".

That's a good idea for filesystems. But OpenTimestamps Proofs aren't really "written to". They're created, and then later validated. Also, being cryptographic proofs, my philosophy is the validator should almost always understand them 100%, or not at all, to avoid any false proofs.

That's also why I picked a binary encoding: it's difficult to parse an OTS proof incorrectly. An incorrect implementation will almost always fail to parse the proof at all, with a clear error, rather than silently parse the proof incorrectly.

We use the same for Lightning: even bits for incompatible changes, odd for backwards compatible changes.

I wouldn't waste the first byte on a null; make use the all four bytes for a "fourcc" code. Then have a null byte soon after that somewhere.

Well, like I said, I wanted a 32-byte magic so it'd look nice in hex-dumps. So I had plenty of room.

OTS proof files typically contain lots of 32-byte hash digests. So "wasting" 32 bytes on magic bytes isn't a big deal.

Reply on news.ycombinator.com