Hacker News Reader

Comment by astrodust | Hacker News Reader

CSV is less a file format and more a hypothetical construct like "peace on Earth".

CSV is a hazing joke we play on noob programmers to demonstrate to them the function that eats a string and a delimiter, and spits out that split string into an array. Then we demo it with the most innocuous plain vanilla csv formatted data we can imagine like '1,2,3' and then haze the new programmer by telling them that's all they really need to know to process CSV files, assuming they can read file lines into strings and manipulate arrays. ha ha noobs ha ha ha. Well its funny if you're not the one cleaning it up or getting the bug reports.

The other side is trolling noob DBAs into thinking they wrote babbys first fisher-price CSV exporter when they ran SELECT CONCAT(columnone, ',', columntwo, ',', columnthree) FROM tb_wtf; and then they get their first bug report about the column data containing a comma, perhaps as a full name.

Personally I think if we're going to haze noobs it would be more civilized to just spank them with a paddle while they repeat "thank you sir may I have another" (And that movie is I believe almost half a century old now) Or do the rituals the navy used to do upon first equator crossing. Tricking noobs into writing CSV processing code is just going to far.

That's not a bad thought. Maybe RFC4180 shouldn't have defined csv, but another similar alternative.

It doesn't so much as it says here are some things that people have done.

> While there are various specifications and implementations for the CSV format (for ex. [4], [5], [6] and [7]), there is no formal specification in existence, which allows for a wide variety of interpretations of CSV files. This section documents the format that seems to be followed by most implementations:

RFC4180 misses some rather famous practical problems, such as until very recently Excel was limited to some 16 bit number of rows, 32767 or 65536, and the bigger and richer the corporation the less likely their software is updated, so you send them a 500K line CSV dump of engineering data, they use Excel as a viewer, and they insta-file a bug report that they are paying good money for 500K lines of data but they checked and we only sent them the first 65536 lines or whatever. Bonus fun if they save the truncated file on itself permanently deleting the excess data. Or if their corporate standard DBMS is Excel, as it often is at larger companies.

This has been fixed, but only in very recent versions of Excel, where I work this was a near daily issue until last year or two.

> This has been fixed, but only in very recent versions of Excel, where I work this was a near daily issue until last year or two.

This has been fixed since Excel 2007. I don't know whether 9 years counts as very recently.

I also agree with the fact that it has been an issue. Even at customers who have upgraded past Office 2007, a lot of corporate forms and standardized data exports are based on old work that was completed before the new Office file formats. Updates to these documents and exports continue to use the old file formats because people make changes to existing artifacts, which are handled in compatibility mode in Office, and no one changes the settings to export .xlsx files.

2007 is still in the future for some enterprise companies.

Absolutely. I am not denying enterprise inertia. But the parent of my original comment specifically said "recent versions of Excel". To me, this sounds like "a version of Excel released recently", not "a version of Excel that certain enterprise clients have only recently started migrating to".

CSV is inferior to TSV, there's really very little reason to keep using CSV other than taking advantage of its ubiquity.

CSV allows commas and tabs in fields (actually, all characters, if you choose to). TSV does not allow tabs.

What's more common, tabs in text content or commas in text content?

I'd suggest you'll see commas far more frequently, such as in addresses and freetext fields.

Can fields with commas be quoted to prevent them from being interpreted as delimiters? Yes they can, but that requires extra vigilance, and in my experience it's frequently overlooked.

As for TSV not allowing tabs... the same type of post-processing that goes into allowing for commas to be quoted can go into replacing tabs with multiple spaces.

> As for TSV not allowing tabs... the same type of post-processing that goes into allowing for commas to be quoted can go into replacing tabs with multiple spaces.

No, replacing tabs with spaces means you cannot distinguish between spaces and tabs.

CSV escaping is completely reversible.

I agree tabs aren't common, but it sure is nice to have your format allow arbitrary data, rather than just the data a particular person prefers.

It's a shame we don't use the ASCII delimiter characters. It would have solved a lot of the quoting issues.

https://en.m.wikipedia.org/wiki/C0_and_C1_control_codes#Fiel...

Agreed, that would be the best delimiter choice for plain text data files.

Heh, for being a hypothetical construct, I sure do get a lot of requests to provide data using it as a format. :)

The "Peace on Earth" feature is in pretty high demand too. Unfortunately, any issues that request this feature end up being closed with WONTFIX by the maintainer.

Reply on news.ycombinator.com