Hacker News Reader

Comment by yan | Hacker News Reader

This especially: "In Python 3, all strings are sequences of Unicode characters. There is no such thing as a Python string encoded in UTF-8, or a Python string encoded as CP-1252. "Is this string UTF-8?" is an invalid question. UTF-8 is a way of encoding characters as a sequence of bytes. If you want to take a string and turn it into a sequence of bytes in a particular character encoding, Python 3 can help you with that. If you want to take a sequence of bytes and turn it into a string, Python 3 can help you with that too. Bytes are not characters; bytes are bytes. Characters are an abstraction. A string is a sequence of those abstractions."

Couldn't have said it better myself.

View on HN

Agreed. If only this had been around 7 years ago when I was first introduced to unicode and character encoding when trying to internationalize a C++ app. It took me forever to get my head wrapped around this idea as I were generally never forced to deal with the actual conversion and Strings were always just Strings. Even 7 years later this document completely cleared up the way I was thinking about it.

"In Python 3, all strings are sequences of Unicode characters"

What does that mean exactly? Everything seemed really well explained until then. The line above lost me.

EDIT: I think I'm confused with the difference between UTF32 and Unicode. Is there one?

Yes. Unicode is a method of converting between code points and characters. A character is a symbol used in a human language or other written communication. A code point is a number, commonly written as "U+<decimal integer expression of the number>".

There are many different ways to actually store these sequences of code points in a computer. UTF-32 is one of those ways. It takes the decimal integer, coverts it to a 32-bit binary integer, and then splits that number into four 8-bit bytes. As the book says, there are problems with space usage -- in ordinary English text, all the code points will be U+127 or less, which leads to a lot of zero bytes taking up space. In addition to the waste of space, zero bytes can cause problems in C, since they're the symbol for the end of the string. So people invented other 'encodings' to convert Unicode code-points into bytes. UTF-16, UTF-8, etc.

Reply on news.ycombinator.com