Recommendations for designing magic numbers of binary file formats
hackers.town
hackers.town
$ hexdump -C foo.ots
00000000 00 4f 70 65 6e 54 69 6d 65 73 74 61 6d 70 73 00 |.OpenTimestamps.|
00000010 00 50 72 6f 6f 66 00 bf 89 e2 e8 84 e8 92 94 01 |.Proof..........|
1) Starts with a null-byte to make it clear this is binary, not text.
2) Includes a human-readable part to make it easy to figure out what the file is in hex dumps.
3) 8 bytes of randomly chosen bytes, all of which greater than 0x7F to ensure they're not ASCII.
3) Finally, a one-byte major version number.
4) Total length (including major version) is 32 bytes to fit nicely in a hex dump.
For most protocols there's only reading and writing, so you can use odd bits to mean "backwards compatible features, you can read even if you don't understand" and even for "stop, we broke compat".
That's also why I picked a binary encoding: it's difficult to parse an OTS proof incorrectly. An incorrect implementation will almost always fail to parse the proof at all, with a clear error, rather than silently parse the proof incorrectly.
OTS proof files typically contain lots of 32-byte hash digests. So "wasting" 32 bytes on magic bytes isn't a big deal.
SHOULD include a zero byte
MUST include a byte sequence that is invalid UTF-8
MUST include at least one byte with the high bit set
Basically so that no valid file in this binary format will be incorrectly misidentified as a text file.
There's no reason for the high-bit "rule" in 2025.
I would argue the same goes for the 0-byte rule. If you use strcmp() in your magic byte detector, then you're doing it wrong
that might be true for ASCII but there are other text encodings out there
And again, if a detector doesn't check for the more specific matches first, before falling back to "ah, that seems to be text", then the detector is broken
I've just dealt with a 7N1 serial link yesterday, they still exist. Granted, nobody really uses them for truly arbitrary data exchange, but still.
These recommended rules cover various common ways to check for text vs. binary, while also aiming to ensure that no genuine text file would ever accidentally match the magic number. The zero-byte recommendation largely achieves the latter (if one ignores double/quad-byte encodings like UTF-16/32).
[0] https://github.com/git/git/blob/683c54c999c301c2cd6f715c4114...
0D 0A 1A 0A -> 0D 0D 0A 1A 0D 0A
But of course this isn't to try and detect a Unix-to-DOS conversion, it's to detect a roundtrip DOS-to-Unix-to-DOS conversion:
0D 0A 1A 0A -> 0A 1A 0A -> 0D 0A 1A 0D 0A
Certainly a very well thought-out magic number.
But the new spec doesn't explain: https://www.w3.org/TR/2003/REC-PNG-20031110/
On the one hand I sort of understand why they don't "If it is not critical and load-bearing to the standard. Why is it in there? it is just noise that will confuse the issue."
On the other hand, it can provide very important clues as to the why of the standard, not just the what. While the standards authors understood why they did things the way they did, many years later when we read it often we are left with more questions than answers.
There are so many random line conversions going on and the detection on what is a binary file is clearly broken.
I don't understand why the default would be anything but "commit the file as is"
Though this has nothing to do with FTP. I’d already abandoned that protocol by then.
Because it’s not uncommon for dev tools on Windows to generate DOS line endings when modifying files (for example when adding an element to an XML configuration file, all line endings of the file may be converted when it is rewritten out from its parsed form), and if those where committed as-is, you’d get a lot of gratuitous changes in the commit and also complaints from the Unix users.
For Git, the important thing is to have a .gitattributes file in the repository with “* text=auto” in it (plus more specific settings as desired). The text/binary auto-detection works mostly fine.
And I agree "commit the file as-is" should be the default - what programming editor can't handle unix newlines?
But go back to the 90s and before, and you’d have to manually select whether you were sending text or binary data. Often these clients defaulted to text and so you’d end up accidentally corrupting files if you weren’t careful.
The pain was definitely real
The file itself has a format that can test a file and identify it (and possibly more useful information) that is read by the file command.
# Various dictionary images used by OpenFirware FORTH environment
0 lelong 0xe1a00000
>8 lelong 0xe1a00000
# skip raspberry pi kernel image kernel7.img by checking for positive text length
>>24 lelong >0 ARM OpenFirmware FORTH Dictionary,
>>>24 lelong x Text length: %d bytes,
>>>28 lelong x Data length: %d bytes,
>>>32 lelong x Text Relocation Table length: %d bytes,
>>>36 lelong x Data Relocation Table length: %d bytes,
>>>40 lelong x Entry Point: %#08X,
>>>44 lelong x BSS length: %d bytes /usr/share/file/misc/magic.mgc: magic binary file for file(1) cmd (version 20) (little endian)
The original sources are mirrored in <https://github.com/file/file/tree/master/magic/Magdir>. The identification of the magic.mgc file itself comes from <https://github.com/file/file/blob/0fa0ffd15ff17d798e2f985447...>.
This isn't a solvable problem. File formats evolve in messy ways, they always have and always will, and "magic numbers" just aren't an important enough part of the solution to be worth freaking out about.
Just make it unique; read some bytes out of /dev/random, whatever. Arguments like the one here about making them a safe nul-terminated string that is guaranteed to be utf-8 invalid are not going to help anyone in the long term.
It is to solve the problem of how to build a file manager that can efficiently recognize all the file types in a large folder without relying on file name extensions.
If you don't include a magic number a file manager would need to attempt to parse the file format before it can determine which file type it is.
You still need a way to make it hard to fool people with deceptive extensions, though, and that's where the magic numbers come in.
Yes it is, though. Does your file manager want to display Excel files differently from .jar files? They're both different "versions" of the same file format! Phil Katz in 1988 or whatever could have followed the pedantry in the linked article to the letter (he didn't). And it wouldn't have helped the problem at hand one bit.
And yes making it obviously binary is helpful, e.g. for Git.
Unpopular corollary: thinking those are two separate actions is a terribly bad design smell. What are you going to do with that file you "identified" if not read it to get something out of it, or hand it to something that will.
If your file manager wants to turn that path into a thumbnail, you have already gone beyond anything the magic number can have helped you with.
Again, needless pedantry. Put a random number in the front and be done with it. Anything else needs a parser anyway.
> Unpopular corollary:
To me at least it comes across as condescending and unnecessarily edgy or contrarian for the sake of it. I am perfectly able to gauge whether a comment is popular or not.
Anything where there's a decision based on the file format but not its contents. This happens all the time.
* Telling the user what file type it is.
* Choosing a parser to load a file with.
* Restricting file types on upload forms.
* Identifying files for linters.
* Associating files with programs.
Ok some of those use file extensions because it's a lot easier and text based formats often don't have a magic number, but it's still a valid use case.
7F 45 4C 46
- MUST be at least four bytes long, eight is better -> check, but only four
- MUST include at least one byte with the high bit set -> nope
- MUST include a byte sequence that is invalid UTF-8 -> nope
- SHOULD include a zero byte -> nope
So, just 1.5 out of 5. Not good.
By the way, does anyone know the reason it starts with DEL (7F) specifically?
And to be fair, including the 4 byte following the magic number make the ELF-format qualify at least 3 out of the 4 'MUST' requirements:
_ 7F 45 4C 46
- 0x04: Either 01 or 02 (defines 32bit or 64bit)
- 0x05: Either 01 or 02 (defines Little Endian or Big Endian)
- 0x06: Set to 01 (ELF-version)
- 0x07: 00~12 (Target OS ABI)
Still not a shiny example though...
When it comes to the "eight is better" requirement, at least Linux does not care what comes after the fourth byte for identification purposes, so I think that does not count either.
> MUST include a byte sequence that is invalid UTF-8
Making the magic number UTF-8 (or ASCII, which would still break the rule) would effectively turn it into a "magic string". Isn't that the better method for distinguishability? It's easier to pick unique memorable strings than unique memorable numbers, and you can also read it in a hex editor.
What would be the downsides?
Or is the idea of the requirement to distinguish the format from plaintext files? I'd think that the version number or the rest of the format already likely contained some invalid UTF-8 to ensure that.
If you really wanted to go the extra mile, you could also include an impossible sequence of UTF-16 code units, but I think you'd need to dedicate 8 bytes to that: two invalid surrogate sequences, one in little-endian and the other in big-endian. You could possibly get by with just 6 bytes if you used a BOM in place of one of the surrogate pairs, or even just 4 with a BOM and an isolated surrogate if you can guarantee that nothing after it can be confused for the other half of a surrogate pair. However, throwing off UTF-16 detection doesn't seem that common or useful; many UTF-16 decoders don't even reject these invalid sequences.
#!/usr/bin/whatever^@^@^@^@^@[HDR]
The header format could allow for variability in the path; the #! and [HDR] part could be enough to give it identify it.
> MUST be the very first N bytes in the file
For every system to be able to parse it without loading the entire file
> MUST be at least four bytes long, eight is better
To reduce risk of two different binary files on the same system having the same magic number
> MUST include at least one byte with the high bit set
To avoid wrongful identification as an ASCII file (ASCII doesn't use the high bit)
> MUST include a byte sequence that is invalid UTF-8
To avoid wrongful identification as UTF-8 text file
> SHOULD include a zero byte
To avoid wrongful identification as ANY text file
> For every system to be able to parse it without loading the entire file
It also solves the ambiguity problem, zip files have the magic numbers at the end, and most other files like pdf have the magic numbers at the beginning, so you can have a file that is both a pdf and a zip file.
As a more concrete example of how file type confusion can bite you, you can imagine a hypothetical photo sharing service that lets users upload both individual images and zip files containing images; The basic structure of the server looks something like
function user_upload_hook(file):
if(is_zipfile(file)):
extract(file, tempdir)
else:
move(file, tempdir/file)
for image in tempdir:
create_thumbnail(image)
...
A malicious user has concatenated a JPEG of a cute kitten with a zip bomb. `file` reports that the uploaded file is a JPEG, so it's fed through unchecked to the server. The application's `is_zipfile()` correctly identifies that the file is a valid zip file, so the application extracts it and DOSes the server. The two different layers of the stack disagreeing on how to classify the offending file directly lead to an exploitable vulnerability.
0xDC 0xDF X X x x (0x01 0x00 | 0x00 0x01)
0xDC 0xDF are bytes with the high bit set. Together with the next two bytes, they form a four-byte sequence that cannot appear in any valid ASCII, UTF-8, Corrected UTF-8, or UTF-16 (regardless of endianness) text document. This is not a perfectly bulletproof declaration that the file does not contain text, but it should be strong enough except maybe for formats like PDF that can't decide if they're structured text or binary.
X X x x: Four ASCII alphanumeric characters naming your file format. Make them clearly related to your recommended file name extension. I'm giving you four characters because we're running out of three-letter acronyms. If you don't need four characters, pad at the end with 0x1A (aka ^Z).
The first two of these (the uppercase Xes) must not have their high bits set, lest the "this is not text" declaration be weakened. For the other two (lowercase xes), use of ASCII alphanumerics is just a strong recommendation.
0x01 0x00 or 0x00 0x01: This is to be understood as a 16-bit unsigned integer in your choice of little- or big-endian order. It serves three functions. In descending order of importance:
It includes a zero byte, reinforcing the declaration that this is not a text file.
It demonstrates which byte ordering will be used throughout the file. It does not matter which order you choose, but you need to consciously choose either big- or little-endian and then use that byte order consistently throughout the file. Yes, I have seen cases where people didn't do that.
It's an escape hatch. If one day you discover that you need to alter the structure of the rest of the file in a totally incompatible way, and yet it is still meaningfully the same format, so you don't want to change the name characters, you can change the 0x01 to 0x02. We both hope that day will never come, but we both know it might.
Maybe a zero, the UUID as ASCII, then another zero, then a human readable description for debugging and search, or a structured metadata header.
But first, ask yourself why you are designing a binary format, unless maybe it's a new media container.
When would someone ever want a binary file that's not zip, SQLite, or version controllable text?
It feels like there’s an infinite number of answers to this, but to choose one: when choosing the format to allow memory mapping makes some operations simpler or more performant?
Occasionally, you also see a file header, followed by a size, and an "x", that often indicates a block of ZLIB compressed data.
> When would someone ever want a binary file that's not zip, SQLite, or version controllable text?
Maybe I'm not getting the humour here, but in case you are being serious binary files do have a few advantages over text formats.
1. Quick detection (say, for dispatching to a handler) 2. Rapid serialisation, both into and out of a running program (program state, in-memory data, etc) 3. Better and safer handling of binary data (no clunky roundtrips of binary blobs to text and back again) 4. Much better checksumming.
It makes sense for model weights and media and opaque blobs where you don't need to load just a part of it, but I see a lot of custom binary save files that don't seem to make any sense.
If it's a server, everything is probably in a database, and if it's a desktop app, eventually something is going to make an 8GB file and it's probably going to be slow unless you have indexing.
People are also likely to want to incrementally update the file as well.
If you're sure nobody will ever make a giant file, then VCSability is probably something someone will want.
Binary formats are ideal for critical applications where you want to either 1) parse the file correctly, 2) fail to parse the file at all. Non-binary formats (and re-use of existing binary formats) tend to have failure modes where you parse the file incorrectly due to a bug, resulting in something bad happening like a security exploit.
Sometimes I like to have fun and encode a 1337-code easter egg in the hexadecimal representation