Hacker News Reader

Comment by IanCal | Hacker News Reader

> Unlike binary formats or database dumps, these files don't hide their meaning behind layers of abstraction. They're built for clarity, for resilience, and for people who like to know what's going on under the hood.

Csv files hide their meaning in external documentation or someone’s head, are extremely unclear in many cases (is this a number or a string? A date?) and is extremely fragile when it comes to people editing them in text editors. They entirely lack checks and verification at the most basic level and worse still they’re often but not always perfectly line based. Many tools then work fine until they completely break you file and you won’t even know. Until I get the file and tell you I guess.

I’ve spent years fixing issues introduced by people editing them like they’re text.

If you’ve got to use tools to not completely bugger them then you might as well use a good format.

View on HN

If you're reading in data, you need to parse and verify it anyway.

Which you might not be able to do after it’s been broken silently.

That's still an issue with binary files too, and you can't even look at them to fix.

Binary files are not usually edited by hand, and should not get broken without a broken parser or writer. There can be many more ways of fixing them too because it may not be as ambiguous as broken sv files.

They're standardized[0], so it's only stupid humans screwing them up.

Maybe you need a database or an app rather than flat files.

0. https://www.ietf.org/rfc/rfc4180.txt

That came far after csv files started being used and many parsers don’t follow the spec. Even if they do, editing the file manually can easily and silent break it - my criticisms are of entirely valid to the new spec files. The wide range of ways people make csvs is a whole other thing I’ve spent years fixing.

It’s not about the stupidity of the humans, and if it was then planning for “no stupid people” is even stupider than those messing up the files.

> Maybe you need a database or an app rather than flat files.

Flat files are great. What’s needed are good file formats.

TOML

What's the problem?

What are you trying to ask? I don’t understand. I’m not talking about toml.

I gave you a good text file format. You're acting like there are no good file formats. Either invent a domain-specific one, use a standard one, or use a different modality rather than complain that a utopia you won't bother to create doesn't exist.

Csv files are bad for many reasons, some of which are listed as positives in the article. I’m not talking about other formats.

But TOML is not a good file format. Quite the opposite actually.

https://hitchdev.com/strictyaml/why-not/toml/

I found that post unconvincing.

> It's very verbose.

This is his example: https://github.com/crdoconnor/strictyaml/blob/master/hitch/s...

I think you shouldn't use yaml or toml for this.

> TOML's hierarchies are difficult to infer from syntax alone

True! The point of TOML is to flatten the hierarchical structures. I would argue your configuration files shouldn't have much nesting anyway.

> Overcomplication: Like YAML, TOML has too many features

Basically TOML has a date type and all associated problems and advantages. I think it's a reasonable thing to include.

> Syntax typing

I think this is a good thing. I want to know whether something is a string or a number.

json makes for a great flat file format these days, with jq around to munge the data in it. csv is pretty bad for errors. mostly use it to dump data when I need to pass it to someone that will want to shove it in excel.

> They're standardized[0]

From that article:

“This memo […] does not specify an Internet standard of any kind”

and

“Interoperability considerations:

Due to lack of a single specification, there are considerable differences among implementations. Implementors should "be conservative in what you do, be liberal in what you accept from others" (RFC 793 [8]) when processing CSV files”

Are you AI? I was replying to a comment, not the article.

Also, you're quoting me to myself: https://news.ycombinator.com/item?id=44837879

Reply on news.ycombinator.com