Hacker News Reader

Comment by mikestew | Hacker News Reader

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-characters.html

View on HN

> One might question why all control codes in the ASCII character set have low values, but the DEL control code has value 127. This is, because this specific character was defined for deleting data on paper tapes. Most paper tapes in that time used 7 holes to code the data. The value 127 represents a binary pattern were all seven bits are high, so when using the DEL character on an existing paper tape, all holes are punched and existing data is erased.

I love this, it shows just how old the roots of ASCII are.

There's a problem with that explanation: the binary representation of "deleted data" has a different meaning than the control code "delete data".

Interesting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as.

Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, separating the actions into two codes meant that the printer would not miss out any printable characters.

(At least, I read that somewhere on the internet and assumed it was true!)

It's more likely that it's because they are two separate physical actions (returning the head to the left, and advancing the paper one line). They could be used independently: You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again.

A carriage-return operation takes much longer than a single character, or even two or three. It doesn't make sense to issue two characters just to take up time. The printers always had to have some internal buffer memory (and handshaking over the communication lines to say when the buffer is full) in order not to lose any characters.

"You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again."

Last I checked, this still works even on laser printers (at least on a LaserJet), when sending data to it as plain text. It's not actually printing over itself, but it knows to make the repeated characters bold.

less (among other unix tools) does this too (but you have to do one character, bs and the character again). There are more, like _, bs, character underlines (like cat there is ul that handles this specifically). If your terminal supports os (overstrike) in it's terminal description it handles that natively.

> You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again.

True, but mildly redundant: "overprinting" was explicitly the purpose of 0x08 backspace (which had nothing, originally, to do with 0x7F deletion.)

To overprint a whole line using 0x08, you'd need one 0x08 for each character in the line. So an N-character line overprinted that way would take N3 characters in memory.

Using CR, you'd need N2 + 1 characters.

I had a daisey wheel printer in the late '80s that had a few characters of buffer. I had to know that it took quite some time for every CR. It would do the niave bold of the full line with CR, but if it got X^HX it would hit the X and then slide the head over a bit to the right to smear and get the bold effect, which looked much better. It was not uncommon and that's another reason a lot code did it the BS way for OS capable HC devices.

Ugh, wish I had checked this after sending it, now it's too late to edit. I think the gist of it is clear, but to make sure:

Overprinted with 0x08: requires 3N characters

Overprinted with CR: requires 2N+1 characters

Even ignoring additional time needed to send those extra characters, it also was a lot faster than those control-H's, and (I guess) caused way less wear on your printer.

That presumes you're going like A^HAB^HB; you could just emit foo, and then strlen(foo) ^Hs.

I wonder how precise the backspacing was - how many times could you print a character and backspace before the head had drifted by a full point?

Yes. Separate Carriage Return & Line Feed date back to Murray's 1901 variant of Baudot encoding, and ASCII was created to standardize the various teletype encodings out there so it inherited this way of doing things.

In the days of mechanical line printers CR by itself allowed overprinting for special effects like underlining and bold.

Depends on your platform. Even today a Unix/Linux line ending is almost always just LF.

See: http://en.wikipedia.org/wiki/Newline

Unix = LF Mac = CR DOS = CRLF

The DOS line endings are inherited from previous systems, and while they precisely convey carriage and paper movement of a printer it can be a pain to deal with today.

Unix = LF Mac = CR DOS = CRLF

Please add to that list also

HTTP = CRLF

http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html

However, I still have to see I single webserver that does not accept LF alone instead of CRLF.

Somewhat ironic that this detail of how teletypewriters work was not carried over into Unix whereas every intricacy of how video terminals worked still is with us today.

Only old Macs. They switched to POSIX-style LF with OS X.

True, but since Mac OSX is already known by most people to be *nix based, at least this audience, I felt like that might just add to confusion. There are certainly other OSs like BeOS that I didn't mention, in part because I didn't think they added to the discussion, and also while having fond memories of BeOS, I honestly don't remember what it used. I want to say it was LF, but I no longer remember.

A much more complete history is detailed here: http://web.archive.org/web/20120213005708/http://www.transba...

Agreed. A few days ago, I learned that spaces are illegal between separators in CSV. For example, `"val1", "val2"` is illegal (it should be `"val1","val2"`). Kind of unintuitive given that in most languages, non-delimited spaces are insignificant.

I don't think CSV is a rigidly-defined format - I'm sure some implementations will happily accept spaces between the comma and the opening quote.

This is one reason why I prefer tab delimited files... The format is pretty simple with few edge cases. There really only one caveat to worry about - fields with tab characters. And that's extremely rare in my field.

CSV on the other hand has a few different variations.

That's one reason I prefer TSV... the other one is that you can't use any of the regular Unix tools reliably on CSVs.

Can you explain how tab is any easier than comma? If you have to deal with escaping a character, then certainly it doesn't matter which character it is? For the general case, that is.

Who said anything about escaping a character? For most datasets, having actual tab characters is rare, especially if you can just replace them with spaces. Same with newlines - the just aren't needed in a lot of data. If you're dealing with user-derived text content, tab delimited files might not be the best choice. However, it's great for tabular data.

With CSV, you have to escape quotes, commas, and newlines. With tab delimited, you only have to escape tabs and newlines - and that's if they can't be sanitized out to begin with.

I mean for the general case. Sure, more datasets may be tab-safe than comma-safe. CSV doesn't need quotes if there's no commas. TSV needs quotes or something if the data contains tabs.

Or, you just escape the tab. \t style. (Well, then you need to escape \\, and \n, etc). Really, it just gets messy at that point. Which is why I try to avoid free text in tab delimited files. It's not much better in CSV if you allow newlines within cells.

The thing that gets to me is that the CSV RFC is over 4000, and is less than 10 years old.

There is RFC 4180, which is probably the best formal description of the format we have.

My own CSV parsers (I have written a few by now) usually parse that as if the space before (or after) the quote wasn't there. It's nonetheless something to avoid when writing CSV files (Postel's law, etc.).

I wonder if a 'whitespace separated values' format would make sense, assuming a) the separator is 'any run of whitespace except for a single space (ascii 32)' b) all whitespace in values is folded to a single space (which covers 99.99% of CSV usage). This would make parsing trivial, visual formatting (e.g. alignment) possible, and escaping separators a non-issue.

Spaces are valid in fields, even at the start or end. Furthermore you cannot start quoting in the middle of a field, which is why this doesn't work. For unquoted fields you can put as many spaces as you like after the separator, but they become part of the field, then.

Parsing and outputting CSV can definitely be a pain, and there are a lot of corner cases. I think that this is certainly one of those times where rolling your own should be avoided if at all possible. I am a big fan of Text::CSV or the Python csv module, but even using Text::CSV have had different behavior from different versions.

Or just google for "ASCII table" and open the first hit:

http://www.asciitable.com/

But your link does have more in-depth explanations (including historical info) for some of the control characters.

> man ascii

Just sayin'

Reply on news.ycombinator.com