Hacker News Reader

Comment by flohofwoe | Hacker News Reader

> "UTF-8" support was often just single byte, so it was little more than ASCII

"Single byte UTF-8" is ASCII. That's one of its most important properties.

> Even now it's common for programs to choke on the optional BOM

And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.

View on HN

Your opinion on the BOM isn't wrong, but it's also not germaine to whether or not you're actually following the spec. The spec is the spec. If you don't like it you can get the spec changed. You don't get to ignore the spec and then claim support. That's not how standards work. "I don't like it," isn't a good explanation.

Otherwise I'd be inclined to fix the spelling error in the HTTP referrer.

> BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing

How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM? It seems like one would have to read until finding a code point that would be illegal under one ordering (but files might not include such a code point).

Similarly, a UTF-8 BOM is a useful flag to distinguish UTF-8 from other text encodings. You are right that the ambiguity goes away if those other encodings do, but people don't want to rewrite their legacy files. Some people don't want to use two bytes for common non-ASCII characters, so they are really attached to ISO-8859 or Windows-1252 or koi8r or whatever. CJK languages have their own encodings that are more efficient for their languages. UTF-8 is great for English speakers, but it's a compromise for everyone else, so they might reasonably want incompatible systems for their own use. UTF-8 BOM is a good "magic" sequence to detect encoding as long as people have non-UTF-8 files.

> How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM?

Simple: switch to UTF-8 as the only encoding standard for sharing text data, keep UTF-32 as 'internal' runtime format for random access to codepoints, and get rid of all other legacy encodings (UCS-2, UTF-16, Extended ASCII with code pages, and all the other region specific encodings that popped up in the 70s and 80s because UTF-8 wasn't invented yet.

This general switch to UTF-8 should have happend in the mid-to-late 1990s (e.g. together with the web becoming popular), and Microsoft alone is to blame for dragging this shit along for the next three decades. If all Microsoft tools would only save text data as UTF-8 starting by the end of the last century, but still support reading all sorts of encodings for a decade or so, the transition would have been finished by 2010. Alas, that never happened.

And tbh, the file size argument for alphabets that don't fit into 7-bit ASCII doesn't really make sense anymore today where images and videos make up the vast majority of data volume.

CJK characters is the one big weak point of UTF-8. It forces them to be 3 bytes large instead of the local codepage which is 2 bytes per character, or UTF-16 which is also 2 bytes per character.

IMHO a 30 percent size increase for sharing text data is really not a problem when most data sharing (by volume) happens via audio, video and image data which is infinitely larger. We obviously have both the bandwidth and storage volume to spare for those 30% increased size for text data which among all the other data would account for a tiny fraction of a percent.

Reply on news.ycombinator.com