Hacker News Reader

Comment by lordnacho | Hacker News Reader

Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data.

I guess this is why we have a zillion formats.

View on HN

ASCII Code 29: Group Separator

ASCII Code 30: Record Separator

ASCII Code 31: Unit Separator

https://theasciicode.com.ar/ascii-control-characters/record-...

Thank you for pointing them out. I've browsed, and searched through that ascii table countless times, usually looking for something in particular, yet never realized that there were such a thing as separators before.

this doesn't directly address the ease-of-editing concern, but if you have csv (or tsv) and want ascii (0x1f, 0x1e) or unicode(U+241F, U+241E) separated (or vice versa), 'miller' [0] is a command line tool that supports converting among these formats with these one-liners:

    mlr --icsv --oasv cat input.csv  # csv to asv
    mlr --iasv --ocsv cat input.asv  # asv to csv

[0] https://miller.readthedocs.io/en/latest/

[1] https://miller.readthedocs.io/en/latest/file-formats/#csvtsv...

You're right, Miller is great for this, and it turns out that Miller already offers Unicode Separated Values (USV).

It makes more sense to me in these days of ubiquitous encoding of in-band data that it would be easier to edit if we just used tabs and newlines. If you need a tab in your data, the standard could support either \t (requiring \\t if you need a literal backslash followed by a literal 't') or %09 or &09; or \x09 or something.

If you require escaping then you may as well stick with csv - better to have the character that needs to be escaped to be common enough that people will run into it during testing.

My actual preference is LTSV. Then all that needs escaping or encoding is a quote since a tab or newline inside the quotes is different from one outside the quotes. And I prefer encoding to escaping generally, since it's easier for someone else to come along and build a fairly naive parser.

> Gotta wonder why the format isn't just a column separator char, a row separator char

That's what the article proposes.

Then you can't edit or view it in a normal text editor, which is part of the appeal of CSV.

Maybe normal text editors should support the separators.

That and the fact that the comma (or semicolon, as I prefer it due to the rare usage) is on every keyboard.

A text editor can be made to handle those separators, but editing or processing the file can be more cumbersome than csv and tsv.

I guess if you use an strange character nobody has on their keyboard, you could also just make it a binary format for efficiency

See my sibling comment.

Reply on news.ycombinator.com