Comment by ajross | Hacker News Reader

Comment by ajross | Hacker News Reader

Unpopular opinion: this is all needless pedantry. At best this gives parsers like file managers a cleaner path to recognizing the specific version of the specific format you're designing. Your successors won't evolve the format with the same rigor you think you're applying now. They just won't. They'll make a "compatible" change at some point in the future which will (1) be actually backwards compatible! yet (2) need to be detected in some affirmative way. Which it won't be. And your magic number will just end up being a wart like all the rest.

This isn't a solvable problem. File formats evolve in messy ways, they always have and always will, and "magic numbers" just aren't an important enough part of the solution to be worth freaking out about.

Just make it unique; read some bytes out of /dev/random, whatever. Arguments like the one here about making them a safe nul-terminated string that is guaranteed to be utf-8 invalid are not going to help anyone in the long term.

Magic numbers aren't for parsing files, they're for identifying file formats.

And yes making it obviously binary is helpful, e.g. for Git.

> Magic numbers aren't for parsing files, they're for identifying file formats.

Unpopular corollary: thinking those are two separate actions is a terribly bad design smell. What are you going to do with that file you "identified" if not read it to get something out of it, or hand it to something that will.

If your file manager wants to turn that path into a thumbnail, you have already gone beyond anything the magic number can have helped you with.

Again, needless pedantry. Put a random number in the front and be done with it. Anything else needs a parser anyway.

Speaking of needless pedantry, why do you start with this?

> Unpopular corollary:

To me at least it comes across as condescending and unnecessarily edgy or contrarian for the sake of it. I am perfectly able to gauge whether a comment is popular or not.

> What are you going to do with that file you "identified" if not read it to get something out of it

Anything where there's a decision based on the file format but not its contents. This happens all the time.

* Telling the user what file type it is.

* Choosing a parser to load a file with.

* Restricting file types on upload forms.

* Identifying files for linters.

* Associating files with programs.

Ok some of those use file extensions because it's a lot easier and text based formats often don't have a magic number, but it's still a valid use case.

The magic number isn't about recognizing specific versions. That's just an added benefit if you choose to add that to the magic number.

It is to solve the problem of how to build a file manager that can efficiently recognize all the file types in a large folder without relying on file name extensions.

If you don't include a magic number a file manager would need to attempt to parse the file format before it can determine which file type it is.

Filename extensions are pretty useful. They were adopted for very good reasons, and every attempt to hide them, pretend they don't matter, or otherwise make them go away has only made things worse.

You still need a way to make it hard to fool people with deceptive extensions, though, and that's where the magic numbers come in.

> The magic number isn't about recognizing specific versions

Yes it is, though. Does your file manager want to display Excel files differently from .jar files? They're both different "versions" of the same file format! Phil Katz in 1988 or whatever could have followed the pedantry in the linked article to the letter (he didn't). And it wouldn't have helped the problem at hand one bit.

I agree. The author could go after msgpack, they don’t have a magic number, but support using the .msgpack extension for storing data in files. Since a magic number isn’t required at all, it shouldn’t be required to be good.

Reply on news.ycombinator.com