I just want to chime in and say that Ned Batchelder is one of the greatest human beings alive.

Or, at least, to me he is. This man helps so many people with nothing in return. He's a regular on some python IRC channels, he has personally helped me so much. He makes difficult concepts easy to understand. I encourage everyone to watch his pycon talks. Start with his talk on loops: https://www.youtube.com/watch?v=EnSu9hHGq5o

The problem isn't that there's no implicit conversion between bytes and unicode anymore. The problem is that almost no code beyond the lowest level interfaces is correctly handling it when you need to use bytes. In this case, filepaths (which are bytes in posix, and no using LOCALE is not good enough) don't work. Other examples of things that are bytes:

- Command line arguments

- Environment variables

- stdin/stdout/stderr

- Files in general

- Many expected-to-be-human-readable fields of popular network protocols

In short, there are many situations where you want to treat a bytestring as a string, not as an array of integers.

If bytes in python 3 had acted like str in python 2 (except for the implicit conversions / comparisons with unicode strings), the situation would be a lot better. As it is, they feel like a second-class citizen designed to discourage use, and as a result are unsupported in most libraries that NEED to support them.

(edited for formatting)

> Read, understand, and be glad that the interpreter isn't trying to "help" any more!

As someone who thinks the Python 3 behaviour is largely the right behaviour (I'm unconvinced the solution used for filenames on POSIX is the right one, nor that assuming stdin/out isn't arbitrary bytes is the right choice), I still have a lot of issue with the Python 2 -> 3 migration. (Note I haven't read the article because it won't load here, nor on archive.is.)

As someone who has dealt with a fair number of codebases migrating over the past decade, I would like to have seen a clearer migration path. The route taken basically asked developers to go from:

    def foo(x):
      return x == b"a"
    print foo(u"a")

The fact this went from printing True (Python 2) to False (Python 3) without there ever being any way to know your codebase was doing this, unless you had tests for all such codepaths, meant it was hard to have confidence behaviour was maintained after porting (and I've worked with enough projects that have used Python for scripting without extensive testing of the scripts, often because they're largely doing I/O).

If Python 2.6/7 had a mode like -b (which warns when bytearray and unicode are compared) that warned when str/unicode are compared, that would already have been a big improvement for the migration path. As it is, people have written tools that do this (unicode-nazi), but then you quickly run into the fact that the Python 2 stdlib does this all the time, making it hard to just try and resolve such comparisons within a Python 2 codebase. (Note Python 3's -b does warn for bytes/str!)

Now, at the same time as the behaviour of u"a" == u"b" changed, Python also changed the return type of (e.g.) os.listdir(). This means if you want to compare a different list loaded from elsewhere, you need to have that list in different types depending on whether you're running on Python 2 or Python 3. In a dynamically typed language, it's hard to make all these changes with confidence that you're actually fixing everywhere.

Someone in the past must have invented a wide encoding unlike the popular transition format. I wonder if using that for some use cases would have been less pain because it would remove the whack-a-mole symptom: nothing would work at all until you did all the work to consume/transcode as appropriate.

I suppose it's moot at this point.