Hacker News Reader

Comment by SoftTalker | Hacker News Reader

The ASCII specification defines characters for separating fields, groups, records, and files, but I've rarely seen them used.

View on HN

That's because anyone can easily make a tab character with their keyboard. No one ever remembers the key combination for those special ascii characters.

If it became popular, all common editors would have an easy way to type them.

This encapsulates my problem with CSVs:

- If I send someone a spreadsheet, they'll open it with a spreadsheet application; Excel, LibreOffice, whatever.

- If I send someone a CSV file, they'll want to open it with a text editor.

Ack, no! Open it with a spreadsheet app, or load it into SQLite, or, best of all, open it with VisiData or some kind of editor designed for tabular data.

https://www.visidata.org/

Actually no - spreadsheets classically choose their own way to interpret CSVs, that's the classic way to get your client to continue to send you support requests.

There's a reason so many tools export to xls instead of csv.

And if you just double click on a csv file to open it in excel (rather than importing from excel), excel will happily corrupt your data. Trim leading zeros in IDs, round large integers, not spot the date column is in us vs european format, etc.

Excel will happily ignore if you use tabs as delimiter and show all the columns as a single column. Atleast LibreOffice tries to figure out and confirms the delimiter before opening CSV.

And if there were an easy way to type them, then people would start using them in strings.

And then they'd have to be escaped.

And we'd be back where we started.

While that's true, the way text editors handle these characters is not standardized, and many may not let you input them. One of the important features of CSV/TSV is that they're relatively easy to edit by hand, and for that you need separator characters that are easy for both text editors and humans to work with.

Personally, since I've discovered the field/group/record/file separator characters in ASCII, I've been using them to concat fields and rows on one-to-many SQL joins. They work great for that purpose since (at least on all the projects I've done this with so far) I can be certain that none of the values in the joined data will have those characters, so no further escaping is necessary. For example, in MySQL:

  SELECT
    i.item_id,
    GROUP_CONCAT(CONCAT_WS(0x1F, f.field_id, f.field_value) SEPARATOR 0x1E) AS field_values
  FROM items i 
  LEFT JOIN fields f ON f.item_id = i.item_id
  WHERE ...

Then split field_values with 0x1E to get each field ID and field value pair, and split each of those on 0x1F. Easy as pie.

Reply on news.ycombinator.com