Hacker News Reader

Comment by saurik | Hacker News Reader

...or you could just use the unicode type in Python 2; like: all of my Python 2 software handled languages in other character sets correctly (and yes: this was stress tested often as I have had many many users from China and the Middle East). There was absolutely nothing wrong with Python 2 and Unicode, and if anything Python 3 just made a bunch of stuff impossible to do correctly (in particular encodings for filenames) until years later (I think Python 3.4 or something is when they finally got their Unicode shit together; meanwhile, Python 2 worked great).

View on HN

Except doing the right thing by default is important, and the reality is that Python 3 code is way more likely to handle Unicode correctly, which is what we want.

People constantly complain, but honestly changes like Python 3 have to happen - the alternative is just another completely new language taking over instead.

The only time I had Unicode trouble with python 2 was when a library thought it had to handle encoding. Instead of just piping in bytes on one end and piping out bytes at the other it failed the moment it saw an ü on a system with german text. Unicode certainly has a place, introducing thousands of system specific error and edge cases in what should be a few lines of utility script is not one of them.

I'm less enthused with the defaults than you are. How many people do you think would correctly handle foreign text in this Python 3 exercise:

Write a Python 3 program that takes two arguments, reads a text file named "input.txt", replaces all instances of the first argument in the text with the second argument, and writes the result to "output.txt". For extra credit, add an option to print to stdout.

My bet is very few people would write that program correctly.

Ok, I'll bite.

This is the most straightforward way I can think of to implement the program you described. I think this is more or less what someone would come up with after glancing at the Python tutorial:

https://gist.github.com/avian2/e3968bf933212ee561410b15495e6...

What is wrong with it? It works as expected with non-ASCII characters in both command-line arguments and in input.txt (Python 3.5.3 on Linux, LANG=en_US.UTF-8). Yes, it doesn't work if you have a file that's not consistent with your locale, but you can't really blame Python for that.

I saved this page from the English Wikipedia as input.txt: https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe

    C:\Users\local\tmp>python --version
    Python 3.7.1

    C:\Users\local\tmp>python foo.py Diet Parliament
    Traceback (most recent call last):
      File "foo.py", line 7, in <module>
        text = f.read()
      File "C:\Users\local\AppData\Local\Programs\Python\Python37-32\lib\encodings\
    cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 107: char
    acter maps to <undefined>

Imagine I wasn't a developer who has spent dozens of hours on fixing encoding problems in Python. Imagine I'm just some person using some software I've been given and this is what I encounter. Can I actually complete my work? Is this a good user experience?

The way I usually put it is that Python 3 shifted its priorities.

Previously, if you were a UNIX-y scripter writing UNIX-y scripts on your UNIX-y OS, the fact that Python just kind of pretended encoding issues would never exist was a help to you. It adopted the same "everything is ASCII, or at most UTF-8 in the ASCII range, and if it isn't I'll break in cryptic ways" approach as most other UNIX-y scripting things.

If you were doing anything other than UNIX-y scripting on your UNIX-y OS, this easily became a huge nightmare in Python 2. Django went through a massive rewrite early in its history precisely because of this, to ensure that encoding/decoding happened at the boundaries and everything you'd work with inside a Django app was already a Unicode string. And I remember what it was like trying to work on the web before that approach, and what the work to fix it was like.

Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths, in order to free the rest of the Python community from the nightmares inflicted by prioritizing the UNIX-y scripters to the exclusion of everyone else. So yes, you have to do more work. Yes, you have to learn that a file path is actually an opaque bag of bytes that may not be in any actual encoding and thus can never properly decode to a string. Yes, you have to learn to use fsencode() and the surrogateescape handler in order not to blow up your scripts.

But I'm OK with that, because it puts the workload on you when you're using such a system, rather than magically trying to fix it for you at the cost of everyone else's sanity. It also means that you have to learn to write those scripts correctly. Which is more work than what Python 2 required, but not the world-ending apocalyptic horror it's usually presented as (and is, again, mostly the fault of UNIX-y systems doing their old UNIX-y things, not the fault of Python).

So, write the program correctly. Show me.

> Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths

Show me how much Python 3 improves this. To expand on the program before, make it a directory named 'input' full of files, and a directory named 'output' to put the processed files in. Print each file name as the corresponding file is processed to indicate progress.

I would be applauding Python if it did make the difficulties with this exercise obvious, but it absolutely does not. The file system APIs return strings, but the strings they return may not be valid Unicode. PEP 383 turned the Python 3 str type into a bag of bytes.

Python tries to sweep encodings under the rug. It makes the encoding a default value all over the place and hides conversions everywhere.

I 100% agree that developers need to think about encodings and handle them in their programs. That's exactly why I hate string handling in Python 3: because rather than making you handle the corner cases, it pretends they don't exist, until they're found one by one by your users.

Python 3 encourages developers to write broken string handling code.

So, write the program correctly. Show me.

You just want to fight someone because you're angry, and I don't do that. No matter what someone writes you'll find a way to argue into it being wrong and then prance around declaring "victory".

(I'd also bet that you probably couldn't do it if I were the one who got to set the evaluation criteria, and you also couldn't pass other "challenges" like writing proper HTTP handling -- the person who gets to grade the challenge always "wins", which is why you want to be the person who grades the challenge)

PEP 383 turned the Python 3 str type into a bag of bytes

PEP 383 provided a way to read certain things -- primarily filesystem paths which can be basically anything -- using an escape mechanism to replace non-decodable bytes with surrogates when decoding to string, which in turn allow losslessly transforming back to the original bag of bytes.

Which is necessary, because there are real filesystems out there that really have paths and names that can never validly decode from any known text encoding. It doesn't turn strings into "bags of bytes"; the resulting str still is an iterable of actual valid Unicode code points.

Python tries to sweep encodings under the rug.

As the saying goes, you can't reason someone out of a position they didn't reason their way into, so I won't try here.

Python 3 encourages developers to write broken string handling code.

Python 3 no longer tries to cover for the random gibberish that's legal to put in filesystem paths, and makes the developer handle it. Are there lots of developers out there who don't realize that filesystem paths can legally contain undecodable garbage? Sure. That's not Python's problem to solve, though; it gives you the surrogateescape handler, and the fsencode helper, and keeps working on things like PEP 538 and PEP 540 to try to give you tools to work around it. But Python can't magically fix the mess that is UNIX locales and bag-of-bytes paths (nothing can, short of burning UNIX down and starting over), and doesn't try to do it for you.

You're right that I'm frustrated. I'm not even upset at anyone in this thead, but from previous discussions. I appologise for carrying that baggage here. Having my morality questioned when I legitimately want to make programs work correctly for foreign languages has left me... emotional on this subject.

Unix has its own problems, but the program avian wrote works correctly on Linux. Most encoding issues I encountered when working with Python 3 were on Windows.

PEP 383 was making the best of a bad situation without breaking the API again. The real mistake was having the functions return strings in 3.0. The operating system APIs should have returned path objects that require an explicit conversion to string with an explicit error handling mechanism.

Python gives you all the tools you need to do this right, but they're easy to unknowingly use in ways that break on corner cases. A well-defined API should guide you towards the correct solution and should make pitfalls obvious.

In any case, I should probably give it a rest. I work hard to make sure my programs do this stuff right, and I suppose that's all I can really do.

Well as you can see the problem is that open without encoding arguments uses the system default settings (in this case cp1252) which is always a terrible idea regardless of how a language represents strings internally.

Honestly it surprises me how simple and easy it is to understand this trace and I'd agree with the statement that this is a good user experience.

It's acceptable for developers, but when I say "user friendly" I mean friendly to end users, who may not even know that the program is written in Python. A better message would tell users in plain language what the problem is and how they could fix it (e.g. by setting a flag or an environment variable).

Good for you, but that isn't how it usually worked. Python 3 has made my life easier, but if you really think "There was absolutey nothing wrong with Python 2 and Unicode", I doubt I'll be able to convince you. So I'll simply say again: thank you Python 3 maintainers and developers, it's a huge help to me.

I was always able to work out my unicode trouble in Python 2 ... eventually.

It's better now, less space to screw up.

We where able to delete so much code when we switched to Python 3. When you integrate with especially Windows system converting unknown text was tricky at best.

It’s not that Python 2 couldn’t do it, but the corner cases and error handling was a pain. Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works.

Python 3 was a rocky transision, but I never understood the negativity and I’ll be happy to see Python 2 die, so our system can go back to having just one Python interpretter.

I think this is largely a second system result. If you want correct unicode handling it's not harder in 2.x than in 3.x. The main reason unicode support was more complex in 2.x for the most part is that many APIs supported working with Bytes and on 3.x that burden is pushed to the API consumer.

> Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works.

You should not guess on 2.x either. On the other hand in 3.x the interpreter sometimes guesses the encoding for you when working with OS data on unix and that has lead to some absurd bugs when it falls into that surrogate handling fallback code: http://click.palletsprojects.com/en/7.x/python3/#python-3-su...

>on 3.x that burden is pushed to the API consumer

So for most people it's better. The same goes for the guessing encoding part. Sure, you really shouldn't, but sometimes you had to and then you end up with chardet all over the place. Python 3 may do the exact same behind the scenes. The thing is that the Python core developer are much more talented than I am, so now I get to not think to much about it and just build on their work. From the "regular end-user developer" it's better.

At first blush you would think so, but I know for a fact that Unicode on Windows is not possible to properly handle below Python 3.4.

Python 3’s Unicode handling has been a huge win for cross-platform deployments in my experience. We run a patched version on 3.3 with some backports from 3.4 on millions of machines, and it is actually possible to support non-Latin filesystem paths, where that was just not possible when we were using 2.6.

In terms of an ecosystem, Python 3 forces casual developers to confront Unicode head on, where 2.x the default is to sweep it under the rug.

I’m extremely grateful that a breaking change was made, even if the execution wasn’t perfect.

Click uses ctypes to retrofit some unicode support on 2.x for what it’s worth.

Reply on news.ycombinator.com