Comment by st_goliath | Hacker News Reader

Comment by st_goliath | Hacker News Reader

I once foolishly thought, I'll write a tar parser because, "how hard can it be" [1].

I simply tried to follow the tar(5) man page[2], and got a reference test set from another website posted previously on HN[3].

Along the way I discovered that NetBSD pax apparently cannot handle the PAX format[3] and my parser inadvertently uncovered that git-archive was storing the checksums wrong, but nobody noticed because other tar implementations were more lax about it[4].

As the article describes (as does the man page), tar is actually a really simple format, but there are just so many variants to choose from.

Turns out, if you strive for maximum compatibility, it's easiest to stick to what GNU tar does and favor GNU extensions over PAX header fields. If you think about it, in many ways the GNU project IMO ended up doing "embrace, extend, extinguish" with Unix.

[1] https://github.com/AgentD/squashfs-tools-ng/tree/master/lib/...

[2] https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5

[3] https://mgorny.pl/articles/portability-of-tar-features.html

[4] https://www.spinics.net/lists/git/msg363049.html

> If you think about it, in many ways the GNU project IMO ended up doing "embrace, extend, extinguish" with Unix.

The "extend" phase from the Microsoft playbook is always something that competitors cannot duplicate. Either for legal reasons (patents) or technical reasons (having the "spec" depend on all of the Windows ecosystem).

I don't think that's what GNU did.

No, this was pretty explicitly the reason GCC was not permitted to have independent front- and back-ends like llvm does today. Stallman was concerned someone would write proprietary extensions that way. So, if you wanted GCC, you got the whole GCC, GNU extensions to C and all.

And you only got GCC. There was someone who tried to implement refactoring support for emacs based on gcc, since there already was a llvm based implementation and that obviously wasn't GNU enough. Stallman left him hanging for a year before telling him to stop because it would expose too many gcc internals. He ended up killing the discussion when too many people disagreed with his assertion that text base search and replace was good enough.

GNU extensions are all over the place. The C library has all kind of GNU functions that are more convenient/practical than what Unix had, the command line programs (e.g. coreutils) have feature flags and extensions that the originals didn't have but that made them more convenient to use, and I already mentioned GNU extensions in the tar format.

POSIX aside, Unix-like systems that are still around nowadays and not directly use GNU user space, have copied many of the GNU extensions out of necessity. People have become accustomed to those extensions and written programs & scripts depending on them, so you now need to implement many of the GNU extensions simply to stay compatible.

Even if not consciously done and without malicious intents, this is IMO still fairly similar to how Microsoft played it: add extensions that are convenient, so people use them, eventually rely on them and become a de facto standard.

But as you say, Unix vendors could just implement these features. That's not how Microsoft would have employed its infamous tactic.

I am yet to see Microsoft sue WINE for example.

Any C compiler vendor (even commercial ones) does language extensions, mob => pushing the language forward.

Microsoft does C compiler with language extensions, mob => get your pitchforks.

Similar for vendor lock-in. While very real, it wouldn't be the first time reading topics where the general sentiment is 'oh you poor Apple/MS/RedHat/Visual Studio/... user you're getting locked in, all hope lost' being written by people who don't recognize they are themselves following that principle merely by sticking to one particular OS/editor/debugger/...

I used to be a Solaris fanboy (Linux was only the way I managed to have UNIX at home), and one of my regular tasks as Solaris admin was to install GNU from Solaris Packages repository due to software expectations.

> GNU extensions are all over the place.

On purpose and on accident: when Debian switched its /bin/sh from Bash to dash all sorts of things broke (for us) because GNUism were leaking into what should have been a standard language/feature set.

We had to tell our users: change your code or change your shebang.

> POSIX aside, Unix-like systems that are still around nowadays and not directly use GNU user space, have copied many of the GNU extensions out of necessity.

That reads like a lack of interest to update POSIX to catch up with the de-facto sstandard

I mean, I doubt that GNU would try to block the standardization process to avoid including their extensions.

They pretty much did it with all of the utilities they produced... and I'm glad they did. I don't think it was nefarious, they just thought their extensions made sense at the time and/or filled gaping holes. Even in the early 90's it was painful to go from a machine with GNU stuff installed on it back to a vanilla System V system... they just felt so limited.

That's exactly what GNU did. GNU can take changes from software produced by others under copyfree licenses and wrap it in a copyleft license, then the projects from which it borrowed can't borrow-back the GNU changes because of license incompatibility that runs only one way.

GNU licenses are very much embrace-extend-extinguish licenses.

Perhaps the code itself couldn't be used by other vendors, but the ideas/protocols/interfaces could easily have been implemented by anybody who was so inclined. No trade secrets, no patents, and no dependencies on heaps of legacy cruft were necessary to be compatible. That's an important difference with "Embrace, extend, extinguish".

Ideas, protocols, and (except perhaps in the post-Oracle-Java world) interfaces can still be copied. A lot of open source software actually does exactly that with ideas, protocols, and interfaces from closed source software.

What makes open source software "open source" is the source code, which is subject to copyright law. Ideas, protocols, and interfaces are not subject to copyright; only their expressions are subject to copyright.

The Embrace, Extend, Extinguish approach makes things incompatible with the way others do them to cement hold on a large user base that prevents those users from changing their minds later. Sometimes that takes the form of forcing people to reimplement everything in a cleanroom setting, because they can never keep up that way; that's where the GPL excels within the open source world for EEE. Other times, it's by diverging from norms in ways that others cannot follow without breaking their own ecosystems, which is where the GNU project has excelled with its plethora of subtle incompatibilities.

There are similar problems with csv. A deceptively simple format with many variants to handle various edge cases.

Yes, except it's easy to debug a csv problem because it's generally text in some encoding you can hopefully identify without looking at specific byte offsets in a hex editor: http://fileformats.archiveteam.org/wiki/Tape_Archive#Identif...

In fact, one of the biggest questions I've still yet to answer is whether Docker or OCI have ever clarified just which TAR format must be followed? As far as I can tell, it's a mix of "whatever library X supports"? https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for example.

I'm not sure I want a more complicated format, but I do wish somebody would pick a new file extension for their next use of tar so we could clearly differentiate one tar format from another the next time someone builds on top of simple tar files for their binary distribution, etc.

How many, though?

Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character.

If you have those specified and everyone follows the specification, there shouldn't be any problems, right?

Woe be unto you if you have to process input data where those are not used consistently, of course.

First, there's no specification.

Second, this: http://georgemauer.net/2017/10/07/csv-injection.html

Quote from the site you linked to: "So who’s fault is all of this anyways? Well it’s not the CSV format’s."

And what I wrote before is basically all you need for a specification. Yes, it sucks that there is no fixed one that everyone uses.

> If you have those specified and everyone follows the specification, there shouldn't be any problems, right?

Well, yes, but I feel that that's the point at which CSV definitely earns its "ill-specified format" hat.

Given that there is no way to extract the file specification from the file itself, you can only follow the specification if someone tells you the specification beforehand. You can even use the "wrong" specification and don't notice it because your specification overlaps with the "true" specification in the types of data you're typically seeing (looking at you, optional quotes).

All you need to do is specify all these unspecified things and it’ll be fine.

You forgot the localized decimal separator symbol.

if you start going down that route you will have to specify the format for dates, network addresses, postal addresses, phone numbers...

it never ends

Either you accept that CSV can’t reliably contain fractional numbers or you specify the decimal separator.

CSV doesn't contain numbers at all; it contains strings. How those should be parsed into numbers is not part of the format, unless by CSV you additionally mean "to be imported by Excel". Which is certainly a common use case but not always the case.

By this line of thinking, any file type is specified, because the specification of a file is a sequence of bytes.

Arguably right, practically useless.

The distinction between file format and payload format is perfectly normal and not at all useless.

All file formats have limits for how much they define types for the payload. Most have very limited types. JSON for example knows objects, arrays, strings, numbers, booleans and null. It's not that big a difference to have a format that has only strings.

> Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character.

You also need to specify the separator itself, which despite the name isn't always a comma, but can also be a semicolon or (less commonly) a tab.

The IANA standard for TSV files [1] bans tab characters inside of fields. It's a convenient assumption, so if I'm writing a quick way to sync some data to disk, I choose TSV because the functions can be simpler.

https://www.iana.org/assignments/media-types/text/tab-separa...

To this day I don't understand why standard table formats don't use the standard ASCII unit and record separators.

They are there, they are convenient, and they don't require any visible character sacrifices.

I gues lack of program support. But yeas. That would have been great

Besides inconsistencies, that a non defined quoting and escaping mechanism is already quite painful. Let's look at following single column single line csv:

'\0'

Depending on the source this might be:

- quote back-slash zero quote

- zero

- back-slash zero

- a byte with the numeric value of 0

- quote zero quote

- quote back-slash zero quote

- quote byte-with-value-0 quote

- malformed and should be ignored

Worse the same application might mix different of this quoting mechanisms for different columns or even in the same column. Normally it shouldn't but it's defintely a thing you can find in programs which don't use a proper marschalling library/code, and you know. CSV is easy so you surly don't need to add a library dependency just to use it. (sorry sarcasm).

Or with other words "because CSV is easy" it's somteimes not properly specified and sometimes no "proper" marschalling/serialization library/module is used resulting in ad-hoc fixes for quoting where needed which potentially don't follow any specification either.

The worst part about CSV is the quoting. Oh the quoting. As soon as a human writes something in a plain text field, your nice CSV parsing gets much, much harder.

Salespeople enter things like Company Foo, the One (which used to be Bar). The worst part is that often these are legal names, which means you do need to store them.

When I worked for a FAANG this was one of my major annoyances as there was no canonical number for a customer, so everyone did string matching which broke whenever a company changed their name (like for instance, if they'd just gone public).

Just say no to CSV's with commas in quotes.

Always makes me sad that ASCII delimited was never a thing.

It's almost like they really wanted to avoid this issue back in the 70s so it is built right into ASCII and unicode.

What I like about this is that ending your data with FS means that you can detect file truncation, yet another thing that CSV makes impossible.

I wonder how you embed US/RS/GS/FS within a field though. Prefix with ESC?

How many, though?

As many as there are localized datetime formats :)

there shouldn't be any problems, right?

That would be reallys awesome, yes. With the aid of the other sibling comments you already figured by now it's not as simple, unfortunately.

I suspect that most of the CSV issues I've encountered were produced by ad-hoc writers and parsers embedded in applications rather than serious attempts to implement a CSV library. Behaviors I've encountered during various integrations:

- Values in most columns are escaped, but a few are not. (You have to count commas from both ends of the line and then log a warning and make a guess if there are too many commas.)

- ASCII whitespace is url-encoded (why?) but no other characters are.

- Columns without double quotes can only contain digits.

- Double commas are used as the field separator because they didn't want to implement quoting. (Some of the fields came from user input on a web page, and I prayed for them to hit a double comma and learn the error of their ways, but it never happened. I think their front end developers were sanitizing for them.)

- And my favorite, no quoting was supported at all, double quotes were left unquoted, commas were stripped from values when writing, and when reading, fields were compared against a list of mappings like "Acme Inc." -> "Acme, Inc." to restore the commas.

Reply on news.ycombinator.com