Comment by jcranmer | Hacker News Reader

Comment by jcranmer | Hacker News Reader

As this rebuttal points out, the original article basically finds two flaws in Python 3 and then tries to puff them up as much as possible. These two complaints are:

* Python 3 is not backwards-compatible with Python 2.

* Byte strings and character strings are different things so now I have to care about stuff.

Although I say that it's two things, it's really one thing since the main element of incompatibility is the byte string/character string issue, since it's that incompatibility that makes it hard to know what the right thing to do is.

Having had to deal with internationalization before, I can honestly say that there's only one way to handle it: convert all string data to your internal representation (be it UTF-8 (Rust), UTF-16 (Java), or UTF-32 (Python)) at the margin where you read/write it, and keep binary strictly separate from textual in terms of representation. This is what Python 3 does and what Python 2 doesn't do. I have seen some people argue that Python 2's model is better because it means that writing something like cat is harder, but that's optimizing for the wrong thing. I've seen a build system written in Python 2 break for the sole reason that I thought to write 一 in the commit message.

As for the complaint that it's taking 9 years for Python3 to slowly become the default instead of Python2, keep in mind that 15 years after the release of C99, gcc was still using C89 by default (gcc 5 skipped C99 and went straight to C11).

> Having had to deal with internationalization before, I can honestly say that there's only one way to handle it: convert all string data to your internal representation (be it UTF-8 (Rust), UTF-16 (Java), or UTF-32 (Python)) at the margin where you read/write it

I'm not going to go into the actual subtleties here but there is a lot more to it than what you suggest here. However the biggest mistake you made here is to equate that any encoding is equivalent here. There is a tremendous difference between using utf-8 as internal encoding or a variable encoding like in Python 3. Rust and Go have free bytes/string conversion wheres Python 3 has none of that.

Python 3's string types are internally ASCII (or Latin-1, I'm not sure), UTF-16, or UTF-32 depending on the contents of the string, but the API effectively assumes that the encoding is UTF-32.

The main point is that you don't want to be passing around binary data internally and expecting people to decode charsets as needed--the knowledge of charsets will usually be dropped.

> Python 3's string types are internally ASCII (or Latin-1, I'm not sure), UTF-16, or UTF-32 depending on the contents of the string, but the API effectively assumes that the encoding is UTF-32.

That is incorrect. The API assumes the string is O(1) indexable. It makes no assumptions or guarantees about the internal encoding.

> The main point is that you don't want to be passing around binary data internally and expecting people to decode charsets as needed--the knowledge of charsets will usually be dropped.

This is an incorrect blanket statement and I can give you plenty of examples where this is exactly what you will have to do because the knowledge of the encoding is not available until later. This is in fact a common problem that Python 3 has because it decodes to unicode so early that it had to come up with surrogate-escapes to roundtrip bytes in case the encoding it guessed/inferred was wrong.

He's talking about CPython, the default interpreter/compiler for Python in many systems. CPython internally uses 1-byte representation if the maximum Unicode code point of its chars is 127 or less (therefore ASCII), 2-byte representation for values less than 32767 (thus utf-16, but only the first half-page), and for the rest it uses UTF-32, which means all lookups can be done in O(1).

Assuming UTF-8 encoding for source files is not that strange, as it is getting more and more the default or preferred encoding for string-based I/O.

I understand how CPython works however the external API makes no guarantees about any internal encoding. The implementation of the storage also changed many times.

Do you have a source for Go having free byte/string conversion? To my knowledge it is a copy. Or maybe I don't get your meaning?

I really like the way Go's strings work. I wish Python's strings worked like this too.

https://play.golang.org/p/a70FEDOxBz

> Do you have a source for Go having free byte/string conversion? To my knowledge it is a copy. Or maybe I don't get your meaning?

You can access bytes in go out of the string without even having to perform a conversion. For bytes to string you can use the unsafe and reflect package to create strings from bytes without a copy. (You will need to verify utf-8 integrity of course)

Go strings can contain arbitrary binary data, they are not limited to valid utf-8.

Oh I see what you mean now.

> For bytes to string you can use the unsafe and reflect package to create strings from bytes without a copy.

Super unholy :)

> * Byte strings and character strings are different things so now I have to care about stuff.

The vast majority of Python programmers who don't live in a mostly-ASCII world just breathed a sigh of relief. Python 3 makes Unicode awareness mandatory instead of optional; that's the right move for a web-facing, human-facing language.

In contrast to what Zed suggests, I would argue that the text/binary dichotomy that you describe is actually very important for beginners to learn. Until I learned Python 3, I never really understood how text encodings worked, which confused me whenever I had to deal with Unicode (I'm looking at you, UTF-16).

Reply on news.ycombinator.com