Hacker News Reader

Comment by ahoge | Hacker News Reader

> what's wrong with [CSV]?

The encoding, line endings, and the used separators are random. Encoding/decoding behavior may be locale-dependent. E.g. if you use Excel with a de-DE locale to create a CSV file, Excel with a en-US locale won't open it. The former uses ';' as separators and the latter expects ','.

Since the format is so (seemingly) simple, there are millions of ad hoc implementations out there. Naturally, most of them are horribly broken.

Fun fact: ASCII actually reserved control characters for this stuff. 1F is the "unit separator" and 1E is the "record separator". There is even a "group separator" (1D) and a "file separator" (1C).

http://en.wikipedia.org/wiki/ASCII#ASCII_control_code_chart

View on HN

>Fun fact: ASCII actually reserved control characters for this stuff. 1F is the "unit separator" and 1E is the "record separator". There is even a "group separator" (1D) and a "file separator" (1C).

Which means you can't safely use it for arbitrary data (since your records themselves could contain these separators). Most of the time that doesn't matter; sometimes it does.

You can't use these control characters in CSV either. The grammar from RFC 4180 excludes the 0x00 to 0x1F range.

We need the equivalent of CommonMark. CommonCSV anyone?

We have that already, RFC 4180: https://tools.ietf.org/html/rfc4180

The problem is adoption. Most CSV parsers accept a superset of RFC 4180, because the only thing worse than not adhering to a good spec is tripping over CSV data in the wild.

RFC 4180 does not specify the character encoding.

This isn't a problem with JSON, XML, ASN.1, Protobufs, or Thrift. Anything other than CSV, really.

Indeed it does, although not directly. From the RFC:

> TEXTDATA = %x20-21 / %x23-2B / %x2D-7E

So, it is a subset of ASCII whose hex codes must lie in the ranges given above. Basically, any printable ASCII character with the exception of 0x22 (= double quote) and 0x2C (=comma) which serve other purposes.

From "3. MIME Type Registration of text/csv":

  Common usage of CSV is US-ASCII, but other character sets defined
  by IANA for the "text" tree may be used in conjunction with the
  "charset" parameter.

Basically, you can specify that it's UTF-8 or whatever via the Content-Type response header. Setting this response header isn't something an encoder could do. A CSV parsing library also shouldn't be responsible for loading the CSV file via HTTP. In most cases, CSV files aren't directly fetched via HTTP anyways.

Anyhow, ASCII is utter nonsense. We aren't living in an ASCII world. Even the Americans don't do that. An ASCII-only format is completely useless.

Reply on news.ycombinator.com