Hacker News Reader

Comment by BoingBoomTschak | Hacker News Reader

Can I rant about ASCII FS-US not being used as intended? Because I sure want to every time I must interact with CSV/TSV.

View on HN

Yes and you can add me to the group.

FS is Field Separator. It’s ascii code 0x1C. Theres a whole group of these separators in ascii:

> The separators (File, Group, Record, and Unit: FS, GS, RS and US) were made to structure data, usually on a tape, in order to simulate punched cards.

https://www.ascii-code.com/character/%E2%90%9E

You'd still need escaping/encoding of those characters since they can appear in a record.

There is no escaping of this fact, if you have a storage format you need to be able to store records of that format within the format, so you need some form of encoding/escaping.

It doesn't change much which character we choose for TSV/CSV, tabs, commas, special purpose ASCII/unicode control codes. We still need escaping to store discrete text records within other text records.

But most of the time, you don't. The vast majority of CVS use-cases don't need to store CSV inside CSV. Most of them need to store numbers and text, both of which contain commas and may occasionally contain tabs, but will never contain a field separator character.

"Most of the time" does not mean much. If you need to handle those cases then you need to handle them, regardless of how often they appear. It's not like we'd use a JSON parser that said "yeah, nobody nests 10 arrays, so lets not handle that edge case".

Data transfer formats are not the place to cut these corners so it really doesn't matter what is "most of the time" except perhaps for performance. For correctness relying on "most of the time" is a footgun, it is either always happening or never happening.

When defining a format, you can absolutely set limitations. If most use-cases can use a simple format with this limitation, they can, and the rest can use the complicated one. Data formats don't have to be 100% universal.

Absolutely, but the format described is neither TSV or CSV. That's perfectly fine, but don't call it CSV or TSV if it does not handle the specified ways that those work.

100% agreed. I'd call it Field Seaparator Separated Values - FSSV

Reply on news.ycombinator.com