A rant about Ruby 1.9 String encoding

  > It's a +String+ for crying out loud!  What other
  > language requires you to understand this
  > level of complexity just to work with strings?!

But the rest of the world works differently.

Data can appear in all kinds of encodings and can be required to be in different other kinds of encodings. Some of those can be converted into each other; some Japanese encodings (Ruby's creator is Japanese) can't be converted to a unicode representation for example.

Also, I'm often seing the misunderstanding that "Unicode" is a string encoding. It's not. UTF-(8|16) is. Or UCS2 (though that one is basically broken because it can't represent all of Unicode).

Nowadays, as a programming language, you have three options of handling strings:

1) pretend they are bytes.

This is what older languages have done and what ruby 1.8 does. This of curse means that your application has to keep track of encodings. Basically for every string you keep in your application, you need to also keep track what it is encoded in. When concatenating a string of encoding a to another string you already have that is in encoding b, you must do the conversion manually.

Additionally, because strings are bytes and the programming language doesn't care about encoding, you basically can't use any of the built-in string handling routines because they assume each byte representing one character.

Of course, if you are one of these lucky english UTF-8 users, getting data in ASCII and english text in UTF-8, you can easily "switch" your application to UTF-8 by still pretending strings to be bytes because, well, they are. For all intents and purposes, your UTF-8 is just ASCII called UTF-8.

This is what the author of the linked post wanted.

2) use an internal unicode representation

This is what Python 3 does and what I feel to be a very elegant solution if it works for you: A String is just a collection of Unicode code points. Strings don't worry about encoding. String operations don't worry about it. Only I/O worries about encoding. So whenever you get data from the outside, you need to know what encoding it is in and then you decode it to convert it to a string. Conversely, whenever you want to actually output one of these strings, you need to know in what encoding you need the data and then encode that sequence of Unicode code points to any of these encodings.

You will never be able to convert a bunch of bytes into a string or vice versa without going through some explicit encoding/decoding.

This of course has some overhead associated with it, as you always have to do the encoding and because operations on that internal collection of unicode code points might be slower than the simple array-of-byte-based approach.

And whenever you receive data in an encoding that cannot be represented with Unicode code points and whenever you need to send out data in that encoding, then, you are screwed.

This is a defficiency in the Unicode standard. Unicode was specifically made so that it can be used to represent every encoding, but it turns out that it can't correctly represent some Japanese encodings.

3) Store an encoding with each string and expose the strings contents and the encoding

This is what ruby 1.9 does. It combined methods 1 and 2: It allows you to chose whatever internal encoding you need, it allows you to convert from one encoding to the other and it removes the need to externally keep book of every strings encoding.

You can still use the languages string library functions because they are aware of the encoding and usually do the right thing (minus, of course, bugs)

As this method is independent of the (broken?) Unicode standard, you would never get into the situation where just reading data in some encoding makes you unable to write the same data back in the same encoding as in this case, you would just create a string using this problematic encoding and do your stuff on that.

Nothing prevents the author of the linked post to use ruby 1.9's facility to do exactly what python 3 does (of course, again, ignoring the Unicode issue) by internally keeping all strings in, say, UTF-16. You would transcode all incoming and outgoing data to and from that encoding. You would do all string operations on that application-internal representation.

A language throwing an exception when you concatenate a Latin 1-String to a UTF-8 string is a good thing! You see: Once that concatenation happened by accident, it's really hard to detect and fix.

At least it's fixable though because not every Latin1-String is also a UTF-8 string. But if it so happens that you concatenate, say Latin1 and Latin8 by accident, then you are really screwed and there's no way to find out where Latin1 ends and Latin8 begins.

In todays small world, you want that exception to be thrown.

Conclusion

What I find really amazing about this complicated problem of character encoding is the fact that nobody feels it's complicated because it usually just works - especially method 1 described above that has constantly being used in years past and also is very convenient to work with.

Also, it still works.

Until your application leaves your country and gets used in countries where people don't speak ASCII (or Latin1). Then all these interesting problems arise.

Until then, you are annoyed by every of the methods I described but method 1.

Then, you will understand what great service Python 3 has done for you and you'll switch to Python 3 which has very clear rules and seems to work for you.

And then you'll have to deal with the japanese encoding problem and you'll have to use binary bytes all over the place and have to stop using strings altogether because just reading input data destroys it.

And then you might finally see the light and begin to care for the seemingly complicated method 3.

Sorry for the novel, but character encodings are a pet-peeve of mine.

What Ruby 1.9 gets absolutely right is that its String implementation is completely encoding agnostic (by which I specifically mean that it doesn't force your data to be encoded in a particular way). There are encodings for which there is no safe UTF-8 roundtrip (you can successfully convert the data to UTF-8 nicely, but when you convert back to UTF-8 to that encoding, you won't get the original input back; you'll get a slightly different output).

Rubyists in Japan don't have the luxury of dealing with Unicode all the time; they still get lots of data in ShiftJIS and other encodings. (The same is true of Rubyists elsewhere, but since US-ASCII is a proper subset of UTF-8, most folks don't know the difference; Win1252 is a pain in the ass, though.) If you have to do ANY work with older data formats, you curse languages that force you to use UTF-8 all the time instead of letting you work with the native data.

Most developers don't think about i18n nearly enough in any case; there's a lot more to worry about that simply using Unicode doesn't solve for you. Even the developers of Ruby have to worry about the fact that LATIN SMALL LETTER E WITH ACUTE (U+00E9) is the same as LATIN SMALL LETTER E (U+0065) COMBINING ACUTE ACCENT (U+0301); it doesn't begin to address the capitalization of 'ß' ('SS', which isn't necessarily reversible) or that in Turkish 'ı' capitalizes to 'I', but 'i' capitalizes to 'İ'. Don't EVEN get me started on number formatting...

EDIT: Added the last paragraph.

If you think the way 1.8 handles (or doesn't handle) encoding is just fine, try things you typically do, but with a different language.

For example:

  ruby -ryaml -e'p YAML.dump("こんにちは！")'

IMO, not having an encoding associated with some text sucks if you're a non-english speaker.

The main (runnable) documentation file is here: http://github.com/candlerb/string19/blob/master/string19.rb

All in all, this is a huge transition which will take a while to propagate through the hole Rails stack.

This is not a trivial screwup! This is the sort of screwup that should make everybody who's using that wretched platform think thrice before continuing to use it.

It's a pretty straightforward tradeoff. Of course people who are not Japanese will naturally be upset to pay a cost in complexity for a feature of benefit primarily to a programmers from a single country. Non-Japanese Ruby programmers will just have to decide whether their solidarity with Japanese programmers outweighs their personal and collective inconvenience.

http://github.com/candlerb/string19/blob/master/alternatives...

The first suggestion seems like the logical solution to me however I don't need to deal with this stuff on a day-to-day basis...

This is why most C++ teams prohibit their members to overload operators.

There's a hidden "it turns out that" in this sentence.