Hacker News Reader

Hacker News

Comment by bayesian_horse | Hacker News Reader

Full thread

bayesian_horse

You always want CSV, don't you?

View on HN

Well, one of my bugbears is that most open data is released in weird formats.

Take the boston housing dataset as an example:

http://lib.stat.cmu.edu/datasets/boston

This is one of the most popular beginner datasets around. It's often used in tutorials and when experimenting. Does everyone who wants to use that have to write some custom parser? Why? https://csvbase.com/calpaterson/boston.csv is just so much easier

Which CSV though? The format isn't well-defined in practice: https://donatstudios.com/Falsehoods-Programmers-Believe-Abou...

https://www.w3.org/TR/tabular-data-primer/#dialects

> TSV isn't (not) CSV

why? needs elaboration

As stupid as it may sound, "csv" has become a generic term for any single-character delimited line-based file. We deal regularly with a "csv" that uses the pipe character instead of commas, for example.

yeah that's pretty strange, does the file format documentation call it a CSV? In my experience those files are always referred to in their documentation as "pipe-delimited file"

I just export an excel sheet as scv in libreoffice. Opening the file in libreoffice with default settings gave me a pile of crap.

Except when you want to be sure you get what they intended. If it’s just some little throwaway project and you don’t care if the results don’t match up, use csv.

That's actually when I'd rather use something like pickle or feather, maybe even json depending on the usecase, which is easier to parse back.

I've encountered CSV mostly for data transfer among different programs. There sometimes are better options. Sometimes there isn't, maybe even just because CSV is easier and cheaper to implement on both (or multiple) ends.

it's mainly because excel can read csv ootb (mostly). Also, many old UNIX programs read csv or fixed-width for batch record processing.

Yeah as long as you don’t mind Excel changing a few values here and there and pretend translation and internationalization doesn’t exist. But hey, that’s ‘good enough’ so the developers don’t need to bother using a format developed in this century.

Excel is much like an Incel: it has a tendency to mistake something for a date that isn't.

I would generally recommend parquet for the reasons set out here: https://www.robinlinacre.com/parquet_api/

I don't know if I meant that in jest or for real. CSV is a great format, better than some, until it is not...

Reply on news.ycombinator.com