Hacker News Reader

Comment by Tichy | Hacker News Reader

I never understood how to do unicode in Ruby 1.8, though. Or I could never be sure would put it more succinctly. Especially not with Rails, somehow I could not even find anything about that via Google. It seemed to somehow work, but I want to know what is going on - is my stuff utf-8 or not.

View on HN

As I understand it, Ruby 1.8 strings are simply treated as non-encoded sequences of bytes. It has some support for UTF-8, so as long as you can use UTF-8 and ISO-8859-1 (latin1, since the mapping is the same), just think about it as passing around byte sequences and don't need to do a lot of string manipulation, it's not horrible to work with. However, you do need to jump through hoops to use a number of the standard string methods like #length with multibyte characters (often, this means running the the string through #scan(/./mu) before working with it: http://blog.grayproductions.net/articles/bytes_and_character...). Rails also has a UTF-8 handler to help with manipulation of UTF-8 strings with multibyte characters, and I think it's on by default (http://api.rubyonrails.org/classes/ActiveSupport/Multibyte/H...)

I know that Strings are just non-encoded sequences of bytes. But how do I know that/how Rails works? Ie does it use some regular expressions somewhere that might fail, or not fail? When I was new to Rails they added an extra method to strings to convert it to a kind of unicode string, but somehow later that was lost.

As I said, it seems to kind of work, but I would like to be sure what is going on.

In a sentence, Ruby 1.8 Strings are just a set of bytes, so methods like length count the number of bytes, Ruby 1.9 Strings are a set of characters, so length counts characters. This makes a difference when you have multi-byte characters, and want an accurate length, a correct split, reverse etc.

I found this series of articles well worth the time to read and understand: http://blog.grayproductions.net/articles/understanding_m17n

Reply on news.ycombinator.com