Hacker News Reader

Comment by mamcx | Hacker News Reader

Nope, Straightforward Way is like this:

    name:String, date:Date, value:Int
    "Miami", 2021-08-19 11:54:19.721376-05, 2

And making header mandatory.

I think forcing quoting of strings and forcing "," for separation and "\n" for lines. Dates are ISO, decimals use .

That is all.

P.D: This is similar how I done this for my mini-lang https://tablam.org/syntax. Tabular data is very simple and the only problem with csv is that is too flexible and let everyone to change it, but making the "schema" outside from csv is against it purpose.

View on HN

This addresses part of the problem. Don't forget the character encoding, null encodings, separator, quote and escape chars,...

No doubt, you can use csv in a well defined way if there is agreement and specs. The problem is there simply isn't any defacto standard and it won't be able to establish one in this domain...

Btw. : The python intake package allows you to specify metadata.

But is like with json: We are not debating about that details because is a bit more well defined.

For a improved CSV, that minutae must be defined from the start, but also must take in account how is CSV used, and keep it spirit.

P.D: A example on something like this is https://commonmark.org, that codify the rules for markdown in a better defined speck.

Null encoding is entirely uncovered by the CSV format - if you want to communicate nulls you need to have an internal convention - some sort of reserved string. Quoting characters are pretty commonly agreed to be " (aka &quot) and escape characters are technically unnecessary. Deciding to support \\ and \n is a pretty nice thing to do since some producers and parsers will erroneously choke on mid-string newlines but the only character that actually needs to be escaped (the double quote) can just be doubled up as per RFC4180[1]. The indecision on escaping means that the literal character sequence "\n" is very hard to represent in CSV consistently this is an unfortunate malus of C programmers getting all up in the standard.

RFC4180 is an after-the-fact RFC to try and codify common practices in the industry so some tools may refuse to conform to it - but it's really clearly written and like... super short. If you've never read it just go ahead and do so since it lays things out with exceeding care.

1. https://datatracker.ietf.org/doc/html/rfc4180#section-2

I work with CSVs often and we usually just use blanks like

  item1,item2,,item4,,,item7

to represent nulls. What do you think the pitfalls are here? (Sincere question, wondering how we can improve.)

This breaks in systems where an empty string “” != null but I don’t have an elegant solution.

A typical hack is to distinguish an empty field from a field with a quoted empty string. The former is null, the latter an empty string.

Why do you prefer comma for delimiters? I found tabs are less likely to appear within the strings (so less need to quote the strings if a delimiter is found, and thus the files are smaller). And both can be typed with a single keystroke.

The only advantage I would see for commas is if you are editing the file in a text editor that doesn't highlight strings (like notepad) so it's hard to distinguish them from spaces

If the delimiter isn't a comma, it isn't a CSV. I prefer tabs as well, but that's why I ask for a "tab-delimited text file" instead of a CSV.

In locales where comma is used instead of decimal point, CSV fields are separated with semicolons.

So it called "Character Separated Values".

The advantage you mentioned is the one I've seen that has the most benefit. If users are ever modifying the file raw they're going to bork the tab characters - nevermind some developer opening the file who has his editor configured to aggressively convert tabs to spaces.

Tabs are trivial to actually type (so they're much off than, say, FormFeed) but they're difficult to visually distinguish. I also think it's generally a good habit to just force quoting of all fields in CSV to dodge any impending user errors.

Human readable file formats have surrendered space efficiency for legibility and the main attribute they need to focus on, IMO, is defense against human errors.

Simple, elegant, and backward compatible; albeit with degradation, with the existing "standard". I like this approach.

IMHO the point about .csv is that writing a parser is trivial so none of this matters. Many writers bork on certain types of data. No biggie. Yes there are variants, no they are not complicated. Trying to support _all_ variants of something that is not well-defined is a mugs game. Most of the differences are by mistake. Some of the differences can't be fixed in the parser.

If you need csv to work, pick any one interpretation that does work.

Reply on news.ycombinator.com