Hacker News Reader

Comment by im3w1l | Hacker News Reader

I'm gonna argue that the bug is not in "file". The bug is that the workflow uses file, which by it's very nature must be based on heuristics and will get things wrong occasionally.

EDIT: Well file really had a bug in this case, where they didn't detect what they thought they were, but the point stands.

View on HN

I agree that you should use care when using file. My app relied on "file" to identify file types, and in some cases it mis-identified some Microsoft Office files. The problem was that Apple ships a very old magic file with macOS, and when I reported the bug they said they don't care.

Not sure why people think they can rely on the pre-installed Unix utilities on macOS.

EDIT: Apple routinely don't update the Unix utilities for a decade...

Yeah, those idiots! They buy something from Apple, and then find it doesn't work. What did they expect.

This, of course, is passive-aggressive snark. Suffice to say that I disagree with you; why shouldn't people expect software they buy from Apple to work? People buy from Apple because they don't want to have to deal with all this sort of crap; they just want it to work.

I am more surprised Apple doesnt invest in their own Unix utils packages honestly. Its not like they dont have the money to do so.

Steve Jobs originally wanted to strip out the shell entirely and only support POSIX at the level of the C API. To this day, Apple acts like they've been strong-armed into providing support for the Unix utilities.

I can understand the attitude that 99% of their userbase will never touch the command line, but that last 1% represents the developers that keep the whole ecosystem going.

We actually broadly agree that it is a shame that Apple don't support their Unix utilities.

Excel mis-identifies CSV files whose first column is named "ID" as SYLK files.

But who would EVER think of naming the first column of a spreadsheet "ID"?

https://www.optibrium.com/community/forums/6-general-how-do-...

>Q: I have saved the result table from admensa into .csv format. However I am not able to open in excel. It is said that SYLK: file format is not valid. Do you have any experience on it? Thanks.

>A: This is an Excel 'feature.' I would guess that the first column of your data is entitled "ID". For some reason, Excel interprets any CSV file beginning with "ID" as an SYLK format file (not really sure what that is!). If you edit the file in, say, Notepad and change the name of the first column in the header row, you should find that it will load without a problem.

>This will happen if you save a CSV file from any application with the first characters "ID". So, unfortunately, there's not much we can do about it, other than to recommend that users don't call their first column "ID"!

Excel actually mostly ignores the extension and relies on heuristics to detect the file type on load. Many web applications used to implement "XLS export" by writing out CSV file with .xls extension.

This magic detection of file type has one unfortunate side effect: it is not exactly obvious how to convince Excel to load plain text data that is not CSV with delimiters from the current locale. (which is major PITA with most European locales where Excel uses "Semicolon"-SV and not "Comma"-SV but still calls it CSV)

You’ll have to excuse me because I’m not in front of a windows computer right now, but if you open excel first, there is a data tab on the ribbon and an import data wizard or something like that and you can tell it what the delimiter is. You can also apply types to columns and whatnot before importing. Not the quickest but does help extract the data.

This is a fairly new feature in Excel.

The lack of this feature pushed a lot of scientists to OpenOffice/LibreOffice Calc. At first, just to open the file and "Save as Excel", but of course, then they start using Calc.

(At least that's my experience writing scientific software in Europe. Excel localized not just display, but the input and output. With Calc you could just open a file as "pipe separated UTF-8" and be done with it.)

If you rename your .csv as .txt then you get a wizard where you can choose delimiters, etc. This works in Excel 2000. I don't have an older version to hand.

I'm aware of that. But that is also partly my point, before ribbon there was 'File->Import...' which did essentially the same thing but at least for me (and many users I work with) was more discoverable.

Another thing is that LO Calc will present you with text import dialog when you open anything vaguely text like and also when you paste multiline plain text from clipboard.

> Apple ships a very old magic file with macOS

I particularly relish the way it identifies macOS disk image files as VAX executables. I hope they never change that.

Oh yeah, I got bitten by that. Turns out that MS Office files are zip-compressed xml files. Well I didn't know that, and the files were named something like data.csv, so that left me confused until a coworker told me what's up.

I'm not sure to agree: if/as *nix systems do not use the extension to know the filetype, then `file`-like heuristics is the only other way to know the file nature, or do I miss someting?

Presumably OpenOffice could pass along a thing that says "please print this PostScript file" so that the printer software knows which type of file it is printing and doesn't have to guess with "file". OpenOffice knows what type of file it's outputting, after all.

But your point arguably stands: if a user says "print file X", and you want different files printed using different logic (e.g. PDFs and PostScript files should be printed differently than text files), what other option is there other than using something like "file" to try and guess the file type? It would be very annoying for the user to have to specify each time from a massive list of possible optins.

I think the issue here is the question that should be being asked is: "is this a postscript file?". `file` answers a different question, "what is the most likely type for this file?"

There is no perfect heuristic here, because what if you, for example, want to print a plain text file that happens to be PostScript?

In most cases, you can't print plaintext easily to a printer, it needs some format like PS, so your PS file would be converted into a PS script that outputs your PS file.

Are you sure? I am pretty sure a lot of printers support text directly, they use control codes for additional capabilities.

I'm fairly certain (minus Linefeed Printers of the old days) most modern printers are either Host-rendered (ie, the host renders text and tells the printer what to print) or use something like PCL, which even in the most plaintext mode, still requires some setup to describe and format the page.

You can't just pipe a textfile into printer port and hope your printer spits out anything useful.

I remember the bad old days of only 10-20 years ago when it wasn't unheard of to send a document to the printer, then walk over to discover a zillion pages of plain text starting with %!PS-Adobe-3.0.

I'm not sure if there's a different process involved here, but I vaguely remember that a recent network printer I had came with an ftp server that could be used to print, simply by dropping supported files into it. I never experimented with it, though, so I have no idea if it supported simple text files or required PostScript or some such.

It most likely converted incoming files whenever necessary.

In the case of printing, the context provides some information. In particular, Erlang JAM file seems unlikely to be printed.

Well, the true UNIX philosophy is that everything is a text file ...

.. but as one of the other comments says, programs which want specific interpretation should consider having their own heuristics, especially including "precedence". In the Excel example, it should be considering XLS and CSV first before trying other things.

It's both really. A workflow using "file" only (and without -k or so) is silly because, as everybody knows, some files are valid for more than one file format. "file" was obviously wrong in the described circumstances...

I wish filesystems would commonly support to store the Mime type. File extensions seem to me still a residue from MS DOS times. That would also make working with static (file) webservers much easier. (It's such a breeze to work with blob stores that allow to save the mime type...)

Well one advantage of the "file extension" mechanism is that it is the stupidest thing that will work [0]. It's just like having a type code only it is embedded in the file name so it doesn't require any special features from any filesystem.

[0] Actually, UNIX's "everybody just guess" mechanism is probably the stupidest, but it is so stupid I didn't even consider it.

Yes sure, it can always be attached somehow. But even if used everywhere it has it's limits and ambiguities, e.g. when dealing with .jpg vs. .jpeg, .yml vs. .yaml and so on... It gets even more interesting when Editors create a temporary file by adding a prefix or suffix, or git/a package manager creates an additional file for conflict resolution.

File extensions go back a lot farther than MS-DOS, kid. :-)

BeFS does use MIME types.

https://arstechnica.com/gadgets/2001/08/metadata/9/

Yes, this is why you should engineer things properly. Not duck tape them together with shell scripts!

Actually the user was able to solve it because it was in a shell script. If the program had used libfile instead, there would be no easy way to edit it to change the date format other than recompiling it. They would have needed to wait until the maintainers fixed file itself.

using file is a bad idea anyway... you can't rely on a command that performs guesswork (aka "smart"). It's been a cause of security holes in similar contexts (browsers sniffing mime type from file content/extensions)

> duck tape

That's a fascinatingly apt mixed metaphor: duct taping meets duck typing.

Ha I just looked up duck tape on wikipedia and it says "not to be confused with duck typing"!

*duct, ironically.

Duck Tape is the brand name of the original product. "Duct tape" is the generic name that the industry settled on.

("Duck" refers to the water-resistant nature of the tape. Duct doesn't actually mean anything other than sounding like "duck". Ironically, duct tape is the wrong type of tape to use on heating ducts.)

If you're going to be pedantic, then get it right. ;-)

Actually I just looked it up and it's "duck" because it was made from "cotton duck", a type of fabric.

now I just think somebody's getting trolled :D

Wow, I always thought it was duct tape because it was originally made for taping ducts. And that duck tape was just a brand name like gorilla glue. Thanks for that.

Huh, I thought Duck Tape was a NEW brand that came out to capitalize on the generic duct tape. Thanks!

Reading Wikipedia's article has me questioning the information in my above post.

According to Wikipedia, the name comes from it being manufactured from "cotton duck" (not because water rolls off it like a duck). Then a derived product became popular for wrapping air ducts, and the name evolved to duct tape. Then someone trademarked "Duck Tape" and revived the old name.

I'm not sure anymore about the true origins of the terms, but I still maintain that "duck tape" and "duct tape" are both acceptable names (trademark issues aside).

Reply on news.ycombinator.com