Hacker News Reader

Comment by rudolfwinestock | Hacker News Reader

“Any file is a database if you awk hard enough.” —Bryan Horstmann-Allen

https://twitter.com/neilkod/status/914217352564137984

Furthermore, no thread on CSV files can be complete without mentioning this infamous bit of fact-trolling: the ASCII standard has had unit- and record-delimiters baked into it from the beginning.

https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...

View on HN

> the ASCII standard has had unit- and record-delimiters baked into it from the beginning.

That's mad that I've never heard of that before.

I'm amazed this isn't used more often!

They don’t really get used IMO because if you’re storing ASCII or Unicode text, you have to be prepared for those characters to be in your data. And if you’re storing binary data, delimiters don’t really cut it in the first place.

No it's fine. Just base64 encode the binary data. It's common and streams, or you can capture and convert the entire field at once if you need to seek around in the binary data.

https://en.wikipedia.org/wiki/Base64

That only works if the thing consuming your CSV can be configured to decode whatever encoding you picked. Uploading CSVs of data to various SQL servers is an example case where that is often not possible.

I didn't mean for CSV. There's plenty of usecases for tabular data that aren't hand coded that this could be used for. Plenty of times I've been just wanting to STGDD and not having to mess around with a serialization library, or have to select delimeters, encapsulators, do escaping etc.

also they cannot be found in keyboards, if you need to write CSV by hand

You can type it at a terminal the same way you can write other low-ascii codes. For example record separator, 30, is ctrl-^ - you can type it in the terminal by the usual literal control char way, ctrl-v[1].

So for example at a Python prompt:

  >>> ord('^^') # typed ctrl-v ctrl-^ here
  30

(the low unprintable ascii codes correspond to certain ctrl combinations because ctrl-x is just the ascii code of x with the high bits masked off (i.e. & 0x1f) )

[1] custimizable in your terminal settings, see stty lnext ("literal next")

Can't be found, but you can enter them by hand by holding down alt while entering the ASCII code on the numeric keypad. I wonder if they'll survive posting a comment: 28∟ 29↔ 30▲ 31▼ Edit: looks like they do survive. Hmmm... Now I wonder about extended ASCII. └┐│└└╚╦╝

Also, you can leverage the ISO/IEC 2022 extension and represent them with Caret Notation. Note: recommend an escaping mechanism depending on data. REF: https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C0_(AS... An editor such as vim or pager like 'less' will display them. You can also 'cat -v' a file.

Ability to eyeball it on terminal is huge though :-(

Never underestimate ergonomics like screen printability!

If people used them then the characters would almost certainly be represented in terminals as table spacing characters...

Usability suffers, if you can't type the delimiters easily.

I can’t stand it when I’m banging out a CSV by hand and have to manually escape the field and record separators. Happens all the time.

I use TSV for that reason. Most of the time you don't need to escape anything. And from code you need to escape/unescape only 4 chars. \t \n \NULL and \\.

In a terminal, Record Separator is CTRL-^, Unit Separator is CTRL-_. Vi[m] will accept the former literally, and will accept the latter if escaped with a CTRL-V.

oh, I love that! I'm going to use these delimiters from now on.

And they're literally named thus too. US, FS.

We just need an rcat to print them clearly.

Reply on news.ycombinator.com