Hacker News Reader

Comment by habibur | Hacker News Reader

All my data are in TSV files, separated by tabs instead of comma and strings are not quoted. It makes parsing or manually editing data so much easier. You need to escape only like 5 characters \0 \n \r \\ \t and even binary data will fit in there.

I was checking if sqlite's CSV supports custom separator and unquoted strings for this to work. Found it doesn't.

View on HN

This is gonna blow your mind: There are already ASCII (and Unicode) characters for that![1],[2] Have been since forever; the inception of the ASCII standard. So it turns out everyone has been (ab)using commas, tabs, semi-colons, newlines, carriage returns, carriage-return-and-newlines, single quotes, double quotes, quotes around every field, quotes only around text fields, etc, etc... For fifty years -- totally for nothing.

Use the data-delimiter characters actually defined in the ASCII / Unicode specifications to, you know, delimit your frigging data, and you won't have to bother about whether to quote or not to quote, nor about painstakingly "escaping" (and un-escaping) whatever characters you're (mis)using for delimiters.

Sure as Hell blew away at least my mind when I came across it.

(Not that I got to implement it at work, of course.)

___

[1]: https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_...

[2]: https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

I built a desktop app that is based on Sqlite, and it is extended to handle other separators.

Not sure if this is what you are looking for.

You can try it out: https://superintendent.app

PS. it also loads GBs of file in 10-20s because that part is written in C.

I think you can use sqlite-utils to accomplish this if interested: https://sqlite-utils.datasette.io/en/stable/cli.html?highlig...

Interesting I think that makes it the first mainstream real world CSV tool I've used that is truly comma separated values then, rather than delimiter-separated-defaulting-to-comma.

Reply on news.ycombinator.com