Comment by BareNakedCoder | Hacker News Reader

Comment by BareNakedCoder | Hacker News Reader

IMHO, the biggest RTFM-Fail ever is related to ASCII. Think of all the errors made and time lost fixing issues with CSV (comma separated value) or TSV (tab separated files) or pipe-delimited fields in records, etc, etc due to having to handle the case where the delimiter character is not a delimiter character but valid data with a field.

ASCII provided characters specifically for this purpose. But no one seems to have RTFMed. ASCII 31 is a "unit separator" (or field separator as we'd call it today) and ASCII 30 is a record separator. There's even ASCII 29, a group separator, so you can have a set of records related as a group (for example, a group of records of different type but related to a single customer). And there's ASCII 28, a file separator, so you can have multiple "logical" files within one physical file.

What if you want to embed a table using the field separator in a table using the field separator?

There is no fail here, this is what escaping is for. And with escaping, you can nest infinitely.

Escaping drastically complicates parsing (though not as much as quoting, thank god). But you seldom need to nest tables within tables. Much more frequently, table values want to contain embedded newlines.

A general-purpose parser has to be able to handle the possibility of nested tables, so it has to account for escaping. And once you have an escaping-aware parser the choice of delimiter is no longer critical.

Escaping is an unnecessary kludge in 99% of cases. If you need nested tables, then you need a special-purpose format that allows for it--and it will very likely look more like a binary filesystem than a parseable text file. If you don't need nested tables, then escaping is unnecessary if the format uses delimiters which aren't present in the data. A lot of data includes newlines, some data includes tab characters, but only binary data would ever include the ASCII specified delimiters.

Don't get caught in the CS trap of thinking you need a completely "general purpose" parser, when a slightly-less-than-completely-general parser will be significantly easier to write, simpler to understand and debug, and faster to execute. And as mentioned above, if you do find yourself needing more than this parser, then you should be looking at non-parsing based approaches anyway.

I think that's the unfortunate consequence of people without a basic CS background inventing ad-hoc data formats. I recently had to deal with a kind of "nested CSV", where each level of nesting introduced a new delimiter! Who needs recursion?

Luckily people who designed programming languages had a better idea: introducing some begin and end markers can allow nesting without escaping them. Here I have two blocks nested inside of the outer block but I don't have to escape anything: { { } { } }

Some financial protocols use those in anger still. Credit card auths and such. There were some serial things that used them as intended too, transfer protocols etc.

iirc "Relia COBOL" back in the early days of DOS also had a format that used the delimiters and all properly; but then did goofy things like globally translating "'" (apostrophe) to "*" (asterisk) anyway.

Even worse, in CSV the chosen separator may, depending on locale, also be used in number values which do not need quoting.

Reply on news.ycombinator.com