Hacker News Reader

Comment by lubujackson | Hacker News Reader

My heart goes out to anyone that ever has to work with encoding issues. I feel like I know 5% of this stuff and there looms a vast mass of chaos like a Gordian knot whenever I need to dip my toes in it.

View on HN

Every programmer has to deal with encoding; it's a fact of life. A list of bytes without encoding is just that: a list of bytes. Handling bytes as text (I.e.: "I/O")? You are now working with encodings.

And the funny thing is: it's not that complex.[edit: let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful]. Unless you don't know how it works---then it's the mystery Gordian knot, as you describe it.

The irony is that encoding is a worry precisely for those who try and stay away from it.

Don't shy away from encodings; embrace them. Then you will learn to love them.

(Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.)

Python is delightful? Many of the only times i've had issues were in python.

Python3 is delightful.

> let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful

Unless you do multiplatform development, then the language has a hard time saving you (and Python definitely does not)

> (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.)

That's not true at all. A ton of byte sequences (and 13 standalone bytes) are outright illegal in UTF8, there's a fair amount of error handling in a validating UTF8 decoder[0], whereas there usually isn't any invalid byte (let alone byte sequence) in 8-bit codepages or character sets. When you decode random bytes in Windows-1256 or ISO-8859-9[1] and re-encode them as UTF8, the UTF8 encoder isn't the one at fault for your garbage output, as far as I could see it got perfectly valid unicode data.

People passing through unvalidated data (possibly assuming it's UTF8) isn't a problem with UTF8 either, by the way.

[0] whether that's used in strict mode or in replacement mode is a different concern and not a blemish on UTF8 itself

[1] which will always succeed, you can try it at home, just get a bunch of bytes from urandom and feed them to various decoders, chances are low that you'll generate anything the UTF8 decoder will accept, chances are also low that you'll generate anything an ISO-8859 character set will reject.

This article is trying to describe how to work on Windows specifically. I don't think it's saying ignore encodings, though honestly why -everything- isn't just encoded utf-8 I don't know.

It's sad that a "modern" OS that had a mostly ground up rewrite (NT) after utf-8 was invented, doesn't have better support for it. I get in memory storage being utf-16, and I'd even accept modern OSes storing files in utf-16, but utf-8 is so elegantly backwards compatible with ascii, it's brain dead not to fix -everything- to work with it.

Perhaps this is the biggest difference between a closed OS like Windows and it's more open counterparts, if Windows were open then the community could have fixed this issue long ago.

The problem with elegant backwards compatibility with ASCII is that few systems ever stuck to ASCII - everybody used different, incompatible extensions to ASCII, like the ISO 8859 family and codepage 1252. So by being elegantly backwards compatible with ASCII, UTF-8 also manages to be subtly incompatible with the majority of almost-aSCII data that exists, in ways that sometimes don't matter. Until they do.

ISO 8859 (or big brother ISO 10367) with ISO 4873 is fairly sane, backwards compatible with ASCII, distinguishable from UTF-8, and historically supported by X terminals… but not much else. ‘Plain ASCII’ is not often fully supported either; how many people even know that the standard included composing accented characters by overstriking (e.g. a BS " → ä)?

The ASCII standard has nothing to say about backspace overstriking - that would be something defined by a terminal, file format, or wire specification. In a more common example, ASCII has nothing to say about how you move the cursor to the start of a new line. Some terminal standards will do so on an LF, others just move the cursor down on LF; file and wire specs need to take a position on how they want to represent a line break; and now we get to the point where CRLF is a magic sequence needed to trigger a line break in some formats and it is completely unrelated to how a particular terminal behaves.

  > The ASCII standard has nothing to say about backspace overstriking

It seems impossible to get ANSI copies of old standards (even for money) but the ECMA (1973) printing says:

    3.2 Diacritical Signs
    (Positions: 2/2, 2/7, 2/12, 5/14, 6/0, 7/14)
    In the 7-bit character set, some printing symbols may be
    designed to permit their use for the composition of acce‐
    nted letters when necessary for general interchange of
    information. A sequence of three characters, comprising
    a letter, BACKSPACE and one of these symbols, is needed
    for this composition; the symbol is then regarded as a diacrit‐
    ical sign. It should be noted that these symbols take
    on their diacritical significance only when they precede or
    follow the character BACKSPACE; for example, the symbol
    corresponding to the code combination 2/7 normally has the
    significance of APOSTROPHE, but becomes the diacritical
    sign ACUTE ACCENT when preceded or followed by the character
    BACKSPACE.

This is precisely the reason ASCII 1967 replaced ← with _ and ↑ with ^ (explicitly still “circumflex accent” in Unicode) and added ` (“grave accent”, likewise).

ANSI made this optional in the 1986 revision, in §2.1.2 — “The use of BS for forming composite characters is not required.” — with a note that it would likely be removed from a future revision (but there never was another one).

  > In a more common example, ASCII has nothing to say about how you move the cursor to the start of a new line.

It does; it just says something slightly unfortunate about code 0x0A.

  CR  Carriage Return
      A format effector which moves the active position
      to the first character position *on the same line*.

  LF  Line Feed
      A format effector which advances the active position
      to the *same character position* of the next line.

[Italics added] But then it says:

  The Format Effectors are intended for equipment in
  which horizontal and vertical movements are effected
  separately. If equipment requires the action of
  CARRIAGE RETURN to be comhined with a vertical movement,
  the Format Effector for that vertical movement
  may be used to effect the combined movement. For example,
  if NEW LINE (symbol NL, equivalent to CR + LF)
  is required, FE2 shall be used to represent it. This
  substitution requires agreement between the sender and
  the recipient of the data.

  The use of these combined functions may be restricted
  for international transmission on general switched telecommunication
  networks (telegraph and telephone networks).

So CR LF will unambiguously get you the first position on the next line. The code for LF is allowed to be replaced by NL by “agreement”, but CR can't move to the next line.

That's fair, and thanks for digging up those actual standards - they are indeed not easy to find these days. I think though that tends to be the standard speaking to the intended usage of the ASCII codes, more than it does to an attempt to standardize behavior of all input and output devices. It's more a suggestion that some equipment might choose to use backspace for diacritics, and some might separately effect X and Y movement, but others might not. Certainly in practice terminals have always exposed their different capabilities by giving special terminal code meanings to ASCII sequences, and, when terminal capability negotiation was not an option (e.g. in specifying a file format or defining how to separate SMTP headers) ASCII users have made their own standards for these things (I guess this is the sort of 'agreement' the standard alludes to).

Yes, the ‘dual use’ characters were define at the request of European delegations and the language makes it clear that it was expected that English-language terminals would continue to have " look like " and not ¨ and so on. ASCII was defined before video terminals were developed, so no one thought overstriking was remarkable. As it turned out, non-English European language versions ended up not generally using overstrikes anyway, but redefined the ‘national use’ characters #$@[\]{|} instead, which unfortunately did lead to ambiguity.

NT was initially designed (1989-1990) before UTF-8 came along. UTF-16 instead of UCS-2 was shoehorned in later with Win2k.

Realistically, there is no way NT could have used UTF-8 in-memory. UTF-8 was scribbled on a placemat on September 2, 1992 and officially presented in January 1993 (https://en.m.wikipedia.org/wiki/UTF-8#History)

Windows NT was released in July 1993, but development started in 1989 (https://en.m.wikipedia.org/wiki/Windows_NT#Development)

Also, even after UTF-8 was accepted to be a good idea, it wasn't considered a good idea for in-memory usage; indexing UCS-2 encoded strings is way easier, and UTF-16 didn't exist yet. It arrived with Unicde 2 in July 1996 (https://en.m.wikipedia.org/wiki/UTF-16#History)

Yes, the danger is when people think 'Encodings are too complicated! Surely my framework/platform/library should be abstracting all this stuff for me?' and then proceed to assume that because it should be dealt with at a lower level, that is is dealt with at a lower level. All I/O encoding abstractions leak sooner or later. For which reason, all programmers need to stop pretending encodings don't affect them and read this: http://www.joelonsoftware.com/articles/Unicode.html

> let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful

No. In python you have the absolutely retarded behaviour of a program running fine in the command line but crashing if you redirect the output to a file.

Python (the 90% of programs 2.x branch) is horrible. It makes the cardinal mistake of mixing binary data and encoded data in one String type.

C is much better, because at least it doesn't pretend to support encodings. So you're forced to use a 3rd party library anyway.

> C is much better, because at least it doesn't pretend to support encodings. So you're forced to use a 3rd party library anyway.

C is better because it _doesnt_ provide some kind of interface into encoding. A byte is a byte is a byte. Whether that's some slice of a binary blob or the first byte in a multi-byte unicode string is completely irrelevant. The most infuriating thing I've found with higher level languages (Ruby/Python/etc.) is their abstractions on top of string encodings that fail to cover the hundreds of thousands of edge-cases (which, they rightfully shouldn't cover either), meaning that the only time you do end up having to wrangle encodings is when things go south and you have to fight the language's defaults just to get some stupid email address that most likely had a bit flip in transit somewhere to properly write to a CSV.

Python delightful? In Python 2, the behavior of Unicode strings (UTF-16 vs. UTF-32) depends on what options the interpreter was compiled with. Then they made a backward-incompatible change to "fix" Unicode but did the exact wrong thing (UTF-32, UCS-2 or ISO-8859-1 depending on string content) instead of the right thing (UTF-8 storage with UTF-32 iterators).

I got the impression that the internal encoding Python chooses for a string is an implementation detail you shouldn't (and cannot) care about. All the public ways if accessing string contents operate on code points, unless you convert the string to a byte array first.

I've noticed that Excel seems to assert that CSVs are not in UTF8. I know the standard is actually ASCII, but nobody on the team seemed willing to understand why their smart quotes came out as "weird characters" (or even understand what smart quotes are). It came out correctly in LibreOffice.

This is why I always write a BOM in CSV files (yes, even though it's UTF-8 and it shouldn't have a BOM). Excel will actually respect it.

The bigger problem is when Excel tries to be helpful and reformats the data upon opening the file. I can't remember the intricacies, but it used to behave differently whether you opened it from a local path or directly from a browser. I seem to recall using an "incorrect" MIME type helped with that.

Adding a BOM to a CSV file then causes many libraries to add the BOM to the first column's heading name.

Excel predates Unicode by a few years. The first true Excel version on Mac was released in 1985. The first generally recognized use of Unicode dates back to 1988.

Excel CSVs default to the system codepage when reading and writing, which makes some sense given that the files are plain text. You can actually control what codepage is used when importing data but you have to use the import option.

It's an option that just defaults to the system ANSI codepage. After clicking Data > From Text, in the "Text Import Wizard - Step 1 of 3" screen, change the "File origin" to "65001: Unicode (UTF-8)"

Some years ago I had to work with text encoded with 3GPP TS 23.038 [1], which is a 7bit packed encoding with a few quirks. It wasn't particularly fun.

[1] https://en.m.wikipedia.org/wiki/GSM_03.38

The telephony standards are all kinds of bizarre. It's a testament as to how well the specs are written that it actually works.

The specs are incredibly dense and hard to read, but they are quite detailed.

I found many of the protocols to be both incredibly clever and incredibly annoying at the same time.

Agreed completely, I recently had to work with some EBCDIC output files from old mainframe systems. Packed string data in a fixed delimited file. Nail bitingly frustrating stuff.

Having built countless similar conversions, EBCDIC[1] to ASCII can be as simple as a lookup table and "packed string data" is just binary coded decimal [2] with a sign.

What is a fixed delimited file? A 'mainframe' file is generally fixed length or delimited.

[1]: https://en.wikipedia.org/wiki/EBCDIC

[2]: https://en.wikipedia.org/wiki/Binary-coded_decimal

I have used delimiters within a fixed-width file so that I can fit more data in a width field in a fixed-width file layout. This is usually when I have to work with a mainframe, and can't define the layout, of course.

Reply on news.ycombinator.com