Hacker News Reader

Comment by pmontra | Hacker News Reader

The wrong idea was to have file name extensions. The .whatever should should be part of the file name and have no special meaning.

All Unix derivatives look at the first bytes of the file to understand what it is. See man 5 magic for details. The shebang #! is one of those magic values and it makes the OS read the next bytes up to an end of line to get the interpreter and arguments to pass the rest of the file to.

Maybe CPM used extensions to gain some speed (no need to inspect the file), not that hardware running Unix was fast by today's standards. Or maybe to force users to declare the purpose of the file in the name. However, as all unnecessary things it eventually came back to bite people.

View on HN

That's still the wrong approach. You can easily get wrong types from that (e.g. look at the output of "echo FORM > /tmp/mismatch ; file /tmp/mismatch") and that opens up annoying problems for the user in the best case and security issues in the worse case.

The data type of a file is metadata and metadata shouldn't be determined by in-band signaling.

What are you going to do if your system miscategorizes a file? File name extensions are not ideal either as they overload the file name (they are out-of-band signalling regarding the content of the file but in-band regarding the file name) but at least you can change them yourself.

What is the alternative? If you don't trust the information that is sent with the file, do you expect the user to decide on the format of each file?

The file type is an inherent pair of the file content, and must move together. Otherwise your images and documents become only a meaningless set of bytes. The information that "this is a png image" is as much a responsibility of the machine that is sending you the file as the pixel data.

Also, yes, accepting random data from a network is a big security risk. It's up to your OS to handle that data in a secure way, but this is not done by removing the "this is an executable" mark.

The obvious alternative is to have metadata about file in a separate descriptor, which could be saved either in first N bytes of the file, or in the file allocation table itself. However implementing this in this age would be impractical as it would break interoperability with every other (still alive) OS out there.

> have metadata about file in a separate descriptor

Maybe that metadata could be stored in the file's name, like "<filename><separator><format identifier>" or something?

ducks

> However implementing this in this age would be impractical as it would break interoperability with every other (still alive) OS out there.

It's deeper than that, although this is a very good point.

Who teaches the OS what file types exist? How is that updated?

You could rely on the OS vendor, but that's a bottleneck, especially for a consumer-focused OS where most of the applications (and, therefore, most of the application file formats) are not created by the OS vendor, or entities working in concert with the OS vendor. You could have a great new file format and a great new application and be unable to use it, or get anyone else to use it, because the people behind the OS haven't caught up yet, or because the OS updates haven't percolated out yet.

You could trust the application vendors to do it, which is a security hole: The ScuzzyTech Hyper Word Processor really, really wants to open your word processing documents, so it can synergize your monetization with microtransaction-based leveraged technologies. Therefore, all of your Normalcore Text Documents are now ScuzzyTech Hyper Word Documents only to be opened by the ScuzzyTech Hyper Word Processor, because the ScuzzyTech code helpfully updated all of your filetypes and, therefore, file associations.

You and the other people in this discussion are well on the way to reinventing extended attributes, used for exactly this purpose back in the late 1980s and 1990s in OS/2.

I'm all for a metadata standard that would enable filetype independent fields to live and move with your data.

That said, this gives you no security benefit at all. It's basically the same thing we have now, but done correctly.

Well, actually magic(5) is used by the file(1) command to identify a file. So when you type `file foo.jar`, it can tell you "hey, this is a jar file". This isn't something that other applications do, and exec and friends do not look at it to try to determine what kind of file something is or how to execute it.

#! is special - and not because of magic(5) - if a file starts with `#!`, exec and friends will treat it special. Anything else will be attempted to be executed as a binary.

Now, one _could_ go and install binfmt_misc and set it up so that files with a jar's magic number will be executed a certain way (`java -jar ...`), but this is neither out of the box nor something that all Unix derivatives do.

Notably, binfmt_misc will also work with extensions instead of magic numbers.

Doesn’t that get incredibly complicated when dealing with non-binary data? How many bytes would you have to read to determine an html file, a markdown file, an Apache config file, etc. etc.

Reply on news.ycombinator.com