Comment by bruce511 | Hacker News Reader

Comment by bruce511 | Hacker News Reader

Parent Full thread

Ahh, plain text, wherein "plain" does some heavy lifting. If plain text had something as simple as a 4 byte signature it could have been soooo much better.

As it is programs have to guess the following;

A) encoding. If ANSI which code page? If unicode which encoding? In the case of utf-16 big or little endian?

B) line endings? CR? LF? CRLF? I guess no-one uses LFCR... right?

C) number formats? 100.000 or 100,000? Date formats? is that mm-dd-yyyy or dd-mm-yyyy?

D) what human-language is it in?

E) CSV? Don't get me started...

Yes. Text is a lot easier to load, parse, make guesses about than say XLS. Yes all of the above things can be "guessed" to a greater or lesser extent.

Yes BOM exists to at least try solving the encoding question. Of course most files don't use them. Lots of tools don't support them. And they don't solve any of the other issues.

But sure, ASCII files in English with US date formats, and Windows line endings....no problems at all...

ASCII is just a raw binary format. It isn't just for text.

The line endings problem really isn't a problem. Pretty much every text editor out there can handle different line endings.

I don’t think number nor date formats are relevant here. For example you could have that same problem entering text into MS Word. That’s really more of an issue if you want to use text as a database rather than a document format, which isn’t really something that even plain text advocates would generally recommend.

As for human language detection, that’s a much easier problem to solve than decoding a proprietary binary blob.

CSV definitely has its warts. But it’s not like that’s the only plain text option for serialising data. (JSON, jsonlines, YAML, XML, etc). Or you could use the actual ASCII codes reserved for records, if you really wanted something that didn’t require quoting and escaping in plain text. It’s actually a pity nobody does this.

> C) number formats? 100.000 or 100,000?

Most national standards now call for a space (preferably a half width space, Unicode: U+2009) as the thousands separator and then either comma or full stop as the decimal separator.

The half width space was standardised in SI in 1948. See, i.a., https://snl.no/tusenskille

Why is it not 2008 punctuation space to match the width of alternative thousands ,. separator?

I have no idea, you'll have to ask the SI body that made the decision.

I've made a language called CSTML to do some of this: https://docs.bablr.org/guides/cstml

Basically it's a text-based language for embedding semantic metadata into some kind of underlying text stream. Is this interesting to you?

> But sure, ASCII files in English with US date formats

LoL, I love the irony of going with US date formats as the one of the obvious defaults to that variable.

The entire point of plain text is that it doesn’t have number formats, date formats, and so on.

This is the eternal tradeoff of specificity vs. generality.

More generally, it's not rich text. The two terms are a pair, where rich text defines additional structure around the characters in order to specify things like that, and plain text has none of it.

Reply on news.ycombinator.com