Hacker News Reader

Comment by tambourine_man | Hacker News Reader

ASCII has a field delimiter character. The fact that we chose comma and tabs because a field delimiter character is hard to type or see is one of those things that saddens me in computing.

Imagine the amount of pain that could have been spared if we had done it right from the start some 50 years ago.

View on HN

The great thing about comma as a field separator is (1) the character is visible and (2) the character is common, so if there are escaping bugs in either the generator or the parser, they will quickly become apparent. Much better to fail fast than having a parse error at line 28357283 because a more uncommon separator character somehow still made its way into the data.

We have editors that can work with invisible characters. It’s not hard. I do that all the time in Vim with tabs and CR/LF anyway.

Unfortunately that ship has sailed. We have standards for escaping commas, escaping quotes, it’s escaping all the way down

> (1) the character is visible and

This can be fixed in the font

This all predates fonts as we know them today. TTY were limited by the keyboard and the character generator ROM. Today no one thinks twice.

The bad thing is that it is common so you have to escape a lot. The much worse thing is that csv implementations have varying ways of handling escaping, or sometimes don't support escaping at all, so in practice csv files can't be used interoperably.

Unfortunately, ASCII delimiter fields were used streaming data. The pain that created was simple - you'd try to send data containing an RS or a US and it would cause bad things to happen on the other end of the wire.

> Imagine the amount of pain that could have been spared if we had done it right from the start some 50 years ago.

I think it's Putt's Law: If you design something for idiots, someone will make a better idiot. In this case, it took less than 15 years.

I've used the ASCII delimiters in a webapp once; Javascript in the browser formatted data with them and sent it to my server via HTTP POSTs. I was a bit nervous that something in the path would break the data but happily it all just worked fine.

Currently saving the day in a data pipeline project which depends on a tool which only exports unescaped csvs. They work very well through the pipeline, Unix split, awk, and then snowflake all support them nicely. One annoying thing is that they are annoying to type and you never quite know if you need to refer to them using octal, hex or what, and what special shell escaping might be used.

Yeah it's really interesting to me how much of what we use/do is shaped by our input devices. Macropads are a start, but I'd love a keyboard with screens on each key, that's not absurdly expensive and can be layered easily.

Something like the Optimus Maximus?

https://en.wikipedia.org/wiki/Optimus_Maximus_keyboard

(It's been almost 20 years and you still can't get one...)

We'll maybe get there in a roundabout way :

https://en.wikipedia.org/wiki/Sonder_Design

I would think anyone this serious about keyboard control would be able to use layers which is becoming pretty common and not have to "see" the keycaps.

I'm hoping the Flux keyboard will deliver on that.

https://fluxkeyboard.com

Interesting. Are you referring to the unit separator (1F)?

https://www.ascii-code.com/31

Yes, we have unit, record, group and file separators. And we chose never to use them.

It seems as though one could easily build a file format far more useful than CSV simply by utilizing these separators, and I'm sure it's been done countless times.

Perhaps this would make an interesting personal project. Are you aware of any hurdles, missing key features, etc. that previous attempts at creating such a format have run into (other than adoption, obviously)?

The ASCII unit separator and record separator characters are not well supported by editors. That is why people stick to the (horrible and inconsistent) CSV format.

People don’t like invisible hard to type character. They prefer suffering quoting, escaping, escaping quotes and all that fun stuff

Are people actually typing up *SV files by hand? It's trivial to support editing in an IDE and exporting from data-producing applications.

Yes, sometimes, of course. It's a bit like JSON. Sometimes it's easiest to inject a small piece of hand-written data into a test or whatever.

(That said every text editor since ever should have had a "table mode" that uses the ASCII field/record seperators (or whatever you choose), I was always confused why this isn't common. Maybe vim and emacs do?)

I've done ETL work with systems that used the ASCII separators. It was very pleasant work. Not having to worry about escaping things (because the ASCII separators weren't permitted to be in valid source data to begin with) was very, very nice.

I'm a Notepad++ person. When I needed to mock-up data typing the characters was easy-- just ALT and the ASCII code on the numeric pad. It took a bit to memorize the codes I needed to use. Their visual representation is just inverse text and initials.

Unfortunately everyone has moved to "Parquet" (Packet? Parket? Pacquet?) already and we've sailed even further.

I absolutely HATE this Parcquage.

https://www.databricks.com/glossary/what-is-parquet

(I don't think everyone has moved to it. I had never heard of it myself.)

A lot of the machine learning world has started using it, it's annoying as hell, solves a problem that doesn't exist, has inadequate documentation, lacks a good GUI viewer, and lacks good command line converters to JSON, XML, CSV, and everything else.

No binary format will ever kill CSV: plain-text based formats embody the UNIX philosophy of text files and text processing tool pipes to go with them, and nothing is more durable than keeping your data in text based exchange formats.

You won't remember Parquet in 15 years, but you will have CSV files in 50 years.

> You won't remember Parquet in 15 years, but you will have CSV files in 50 years.

You're probably right about CSV but probably not parquet. Parquet is already 11 years old, there are vast data warehouses that store parquet, it's first class in the spark ecosystem, and a key component of iceberg. Crucially, formats like parquet are "good enough" for a use case that doesn't appear to be going away. There is a high probability in my estimation that enough places are still using them in 15 years to be memorable even if it isn't as common or as visible.

CSV is actually a nice format if it weren't for literal newlines being allowed INSIDE values. That alone makes it much harder to parse correctly with simple code because you can't count on ASCII mode readline()-like functions to fetch 1 record in entirety.

Considering it also separates records with newlines, they really should have replaced newlines with "\n" and require escaping "\" with "\\".

I started writing up a spec and library for such a format, then my ADHD drew me to other projects before I finished it. Hopefully I'll get back to it someday.

Edit: this is what I have so far: https://github.com/tmccombs/ssv

What you've got so far looks promising to me. Pretty much just what I was thinking of doing, in fact, albeit with some details worked out that I hadn't yet considered.

Nice job. I hope you come back to finish the project eventually.

The "compact" file format (without the tab and newline) should be the SSV standard for data interchange and storage. The pretty-printed format should only be used locally to display/edit with non-compliant editors, then convert back as soon as you're done.

In time, editors and file browsers should come to render separators visually in a logical way.

Dedicated separator characters don't solve the problem--you'd still need to escape them. Or validate that the data (which may come from untrusted web forms etc.) does not contain them, which means you have another error condition to handle.

Or specify that the data can't contain this data. If it does, you have to use a different format. This keeps everything super simple. And how often are ASCII US and RS characters used in data? I don't think I have ever seem one in the wild, apart from in a .asv file.

I'm no expert on character encodings or Unicode itself, but would this be as simple as checking for the byte 1F in the data? Assuming the file is ASCII or UTF-8 encoded (or attempting to confirm this as much as possible as well), it seems like that check would suffice to validate the absence of the code point in the data, but I imagine it's not quite so simple.

For text data, it would work fine, but you'd have to do some finagling with binary data; $1F is a perfectly valid byte to have in, say, a 4-byte integer.

Without escaping, it wouldn't be suitable for arbitary binary data.

My going assumption is that arbitrary binary data should be in a binary format.

Feel free to correct me, but I figure that as long as data can be from 0x00 to 0xFF per byte, no format that uses characters in that range will ever be safe. I’m not a big C developer but I figure the null terminated strings have the same limitation.

But if its something entered by keyboard you should be ok to use control codes.

Personally, I find tab and return to be fine for text driven stuff. Shows up in an editor just like intented.

The “problem” I’m referring to is that we chose a widely used character as a field separator. Of course you still have to write a parser, etc, it’s just a lot easier if you choose a dedicated character.

There's an ASCII character for escaping, too, if you need it.

The advantage of ASV is not that you can't have invalid or insecure data, it's that valid data will almost never contain ASCII control characters in the record fields themselves. Commas, quotation marks, and backslashes, meanwhile, are everywhere.

I often use them in compound keys (e.g., in a flat key space as might be used by a cache or similar simple key/value store). IMHO, they are superior to other common separators like colons, dashes, etc. because they are (1) semantically appropriate and (2) less likely to be present in the constituent pieces of data, especially if the data in question is already limited to a subset of characters that do not include the separators, which it often is (e.g., a URL).

“Less likely” doesn’t help if you may get arbitrary (user) input. If you can use a byte sequence as the key, a better strategy is to UTF-8-encode the pieces and use 0xFF as the separator byte, which can never occur in UTF-8.

Because they're zero-width. If you can't see them when you print your data, it's a machine-only separator, which makes it a bad separator for data that humans need to look at and work with.

(Because CSV is a terrible data exchange format in terms of information per byte. But that makes sense, because it's an intentionally human readable data exchange format, not a machine format)

Hence https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-u...

I never knew they existed until this post

> ASCII has a field delimiter character.

Where's the key on my keyboard yo make one?

The point of text-based formats is that you can edit them in a text editor by hand trivially, if typing the character is nontrivial, then it entirely defeats the point (that's also why USV ads very little value IMHO).

What’s the key to enter the euro symbol? That means you can’t use it in a text editor?

There is no perfect solution, but I’d rather open a text file in a decent editor than having to deal with the escaping hell that is CSV.

They could have chosen the pipe character “|” at least, but the comma is the thousand separator in many languages (number formatting is kind of important for tabular data, if you ask me) and also, you know, general prose.

>What’s the key to enter the euro symbol? That means you can’t use it in a text editor?

Alt gr+E? Like it's shown on the keyboard.

Not on a US keyboard layout. The point is that we insert characters that aren’t written on the keyboard keys with some regularity, like ©, ®, ™, etc

Well some of us do. There's this interesting effect where many people perceive the limitations on their current tools to be equivalent to limitations on their abstract abilities. If they don't know how to do it, it's impossible.

I think that's exactly the point that the parent poster is trying to make by example? Just because we don't have good tooling today for using ASCII delimiter characters, doesn't mean it's impossible -- just like typing the euro symbol on an american keyboard

Oh yes certainly. And I think that when you're deep into creation it can be really really hard to remember that experience, and so recently I'm trying to find ways to help pull back the curtain for folks.

It doesn't mean it's impossible, but it's definitely cumbersome. Any non English people who has had to type in their native language from an american keyboard can tell you.

Except that not all keyboards have an AltGr key.

European ones tend to have one. US keyboards don't.

Not sure about British. Are they different from US?

> What’s the key to enter the euro symbol?

There's one on French keyboards actually!

And it was there even before we got euro coins in our hands (I know this because I'm still using my first (mechanical) keyboard that I got with my first own PC in 2001: and there is a “€” symbol on it)

There's also the generic currency symbol, ¤, which I think is on some keyboard layouts pre-Euro.

Ooh is that what that is? TIL

That is an underappreciated gem. It should find more use!

Mandatory plug of new Azerty :

https://norme-azerty.fr/en/

Control underscore is the unit separator character. (Some editors may require you to escape that character, though.)

You can actually type a bunch of ASCII control characters very easily on a keyboard. Look at an ASCII table with 32 characters per column (I like this one[1]). The key combo for a control character is Ctrl + the letter on the same row as the control character. So:

BELL Ctrl-G

RECORD SEPARATOR Ctrl-^

UNIT SEPARATOR Ctrl-_

ESCAPE Ctrl-[

You can think of the Ctrl key as clearing the two most-significant bits of the letter's ASCII code. Not all key combos are supported in all environments. Notepad++ doesn't support Ctrl-] (GROUP SEPARATOR) at all, but does support e.g. SHIFT OUT as Ctrl-Shift-N, for instance. The Windows CMD.EXE command line supports many combinations (but not UNIT SEPARATOR, unfortunately), displaying them as e.g. ^[ or ^G in the console.

[1] https://upload.wikimedia.org/wikipedia/commons/1/1b/ASCII-Ta...

If I need a table or Google to figure out how to type something, that's not "very easily"

If you need to train your employees on that, it's not "very easily".

"Very easily" is when I can take any family member who's seen a computer in their life, give them a keyboard and they can figure it out on their own without Google in 2 seconds (like csv).

What I mean is that a simple key combination is easier to use than an Alt code or having to copy and paste from another document. The ASCII table stuff is just fun trivia. "Press Ctrl-_ to insert a column separator" isn't any harder than "Press Ctrl-S to save" or "Press Ctrl-T to open a new browser tab". It's definitely easier than letting your hypothetical family member reinvent character escapes on their own the moment they encounter an address that has an extra comma in it. :-)

An editor could easily have an Insert Special Character menu item.

Reply on news.ycombinator.com