Hacker News Reader

Comment by joefourier | Hacker News Reader

What motivated the Python devs to abandon backwards compatibility with Python 3? I don't see any major departures or revolutionary new features transitioning from 2 to 3. While there's certainly upgrades (e.g. strings), there's enough small changes that upgrading to 3 is rarely a pleasant experience.

View on HN

https://snarky.ca/why-python-3-exists/

What that article casually ignores is the fact that there could have been a Python 2 fix (as Python 3) that took the approach of fixing encoding to utf8 only. If that was the only thing Python 3 vs 2 was breaking, life could have been so much better.

The go approach was not well known at the time, maybe would have worked. Water under the bridge.

Summarizing, that article says "we fixed Unicode". Well, guess what? Unicode in Python3 is a catastrophe.

The truth is that Python3, as a new language incompatible with Python2, was a mistake. I'm sure it was well intended, but here we are.

What's wrong with it?

Consider this, for example:

https://click.palletsprojects.com/en/7.x/python3/

and here, have a nice sandwich of this:

https://stackoverflow.com/questions/57652720/runtimeerror-cl...

or this bullshit:

https://bugzilla.redhat.com/show_bug.cgi?id=1404918

The truth no one wants to hear is that Python2 did Unicode right (or as well as could be expected), while Python3 is a train wreck, even if you just stick to LANG=C and ASCII.

The surrogate escape issue was fixed in 3.7 and you shouldn't use LANG=C as a locale, you should use LANG=utf-8. Python doesn't have bad unicode handling, the system does when you set LANG=C.

And no, for people who actually have to use unicode, python2 didn't do unicode anything like "right". Python2 did unicode "right" for the people with LANG=C and ASCII. Those people are generally wrong.

Even pallets notes

>Misconfigured environments can currently cause a wide range of Unicode problems

The keyword there is misconfigured. Your system is misconfigured. Fix it (by making your locale correct). Then the problems go away.

So, I was right to scorn 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6?

And no, LANG=C isn't wrong. Your software has one job, to at least be correct for LANG=C. If it can't do that, it sucks.

Lang=C is broken if you have non-ascii unicode filenames (or many other things, but those are the most obvious). That's not pythons fault, it's your bad encoding.

Python trusts the system, and the system lies. It's your fault for lying to python. Configure your system correctly, don't lie to your tools and then blame them.

Lang=C is the realm of English only 70s crap programs. You shouldn't use it. Use a modern encoding that supports modern characters.

In addition, the whole idea that a system must have a privileged locale is silly in our connected world. Even assuming a single process (which can be talking to endpoints in many countries) should be locked to a specific locale is nonsense.

The beauty of LANG=C is that it's the only true lingua franca of modern computing. It's a place where all software and programmers can "meet" without privileging any particular locale.

We have 'gettext' and friends for speaking to humans in particular locales. Embedding i18n into core system design just doesn't work.

> Even assuming a single process (which can be talking to endpoints in many countries) should be locked to a specific locale is nonsense.

This doesn't matter. I'm not sure why you think this matters or what problems will be caused, but a server using a unicode locale (C.utf8 or en.utf8) doesn't localize things for those clients, it just is able to not crash when printing their usernames to log files.

> it just is able to not crash when printing their usernames to log files

The usernames could be from a variety of locales. If showing them in that form is seen as valuable, they'll each have to be decoded in their appropriate locale. Using LANG=C doesn't prevent this, it encourages it.

Beyond that, as you point out, Unicode provides lots of new opportunities for crashes. Indeed, one of my favorite ways to crash software (software almost invariably running under a "proper" locale) is to feed it a string of (non-ASCII) Unicode. Boom.

No matter the approach, you must code for correctness and deal with all input. For Unicode, you must always allow for decoding failure.

> The usernames could be from a variety of locales. If showing them in that form is seen as valuable, they'll each have to be decoded in their appropriate locale.

You realize that locales and encodings are different right? En.utf8, c.utf8, and fr.utf8 are all different locales, but are all utf-8 encoded. They'll all work correctly when printed on a machine with any other utf8 locale. The differences arise in things like sort order and regex, where c.utf-8 might be best for a server if you want C or ASCII like functionality.

> Using LANG=C doesn't prevent this, it encourages it.

Well no, because you don't know what encoding each person is using. Either you guess which is error prone, or you have to control the other endpoint so that they include the encoding. But if you control the client, just use utf-8.

Pretty much no software does what you suggest. Go and rust, for example, go further and entirely ignore the system locale and just always use utf-8 for everything. People don't complain.

TIL that there is a "c.utf8". Thank you.

But my God, could this shit possibly be any more byzantine?

(And why doesn't LANG=C just cause c.utf8 to be used?)

POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple.

Trying to wedge Unicode in at this level is a recipe for disaster, and after all these years, Python3 still doesn't have a reliable, coherent story on this.

> POSIX filenames are a series of bytes. That's all you get, and it makes things strikingly simple.

But not windows, or a number of filesystems. Hell the vast majority of the time posix filenames are ASCII, and therefore utf8 compatible.

The rare cases where they aren't break all kinds of other tools because...

> Trying to wedge Unicode in at this level is a recipe for disaster,

Blame every terminal emulator and os in common use. They all display filenames in unicode. So pythons in good company here.

Since you have to display filenames, they need an encoding.

My chosen encoding is usually LANG=C. That is, just show me the bytes, escaped as needed (as with Python 2).

Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘t̵̖̹͙̟͑̈́̎̀͝ͅ ̴̡͒͌̋̾͆̒̎ȧ̴̬̰̪̮̩͉̐̈́̋̇̀̊̕ ̷̫͑́͑̕͠f̸̬̥̼̣̙͎͍͙̤͗͆ͅā̸̛̼̱̠̯͖̫͓͙͒̔͂̅͜͝ͅń̴̨̨̨̦̺̺͗͗̉̎̓̀͘͘͠ ̵̢̖̞͍̻̮͈̘͎̹̉̇̈͐̍́̂̂̀̂o̴̳͙̿͆͝f̵̮̙̞̰͖͉̰̞͕͑̍̀̄̎̍͜ ̵̛̗̙͓̰̹͇̍̒̋͆ͅU̶̧̯͇̰̞̮͋n̵͉̰̰̳̗͎̂͑̉͛̏͛̌i̵̢̢̪͂́̽̀͝͠c̴̢͇͙̙̲̯̫͈̪̀̌̓̂͗̚͜͠ơ̷̬͖̙̯͇̻̱͕͐̎d̶̛̤͉̻͆̋͋̂̀e̵̦̺̹̅͆̀͛̎̎̅.̸̡̟̰͙̮̱͍̼̾͊̑̅͊̚͝

> That is, just show me the bytes, escaped as needed

this isn't actually how python 2 works either. `print("Ń̸̗̮̯͇͈͇̑͒o̸̧̭̯̖̾̐̔̈́̋̈́̈́͘)` will print the same string, since my machine's locale is utf-8. So python2 does the same thing as 3 with regard to machine encoding.

Some time technical debt needs to be eliminated. Keep backward compatability would have meant keeping technical debt. Breaking changes are fine.

The core problem is they didn't break ENOUGH.

Thus Python3 failed (and frequently, continues to fail) to be a compelling upgrade to anyone but unicode purity fetishists.

I am not a "unicode purity fetishists" and python 3 has been a compelling upgrade for me for the past 3 years. I have not written single line of python 2 code since 2 years ago. So your statement that it is not compelling for anyone is wrong. Maybe some people are python 2 fetishists? I have upgraded multiple python 2 codebases to 3 and I am not understanding what people are whining about, it is not rocket science.

This is something people are getting paid to do; maintain their codebase and keep it up-to-date.

Because no one could ever be justified in moving to Python 3 for AsyncIO, f-strings, ordered dictionaries, assignment expressions, pathlib, new features in shutil/tempfile/subprocess/etc, along with massive improvements to memory consumption, runtime performance, and overall stablility?

Until 3.8 py3 was slower than py2 in practically all benchmarks, and 3.8 is a marginal improvement at best. So I’m slapping a massive <citation needed> on your “massive” performance improvements

We switched a service from 2.7 to 3.5 and saw an immediate 45% reduction in memory usage and 20% reduction in runtime. That's massive IMO. We saw further (but smaller) performances benefits upgrading to 3.6, again when upgrading to 3.7, and expect to see even more with 3.8.

https://youtu.be/0kXaLh8Fz3k?t=322

People can and have been fine without those things for a long time. Many of them could have been added to Python2 without any trouble. And yet we still have a GIL.

Was there something about the design of Python2 that would have prevented these features? Honest question.

Probably not, but I’m still glad they did clean up. They could have done more.

>>Thus Python3 failed (and frequently, continues to fail) to be a compelling upgrade to anyone but unicode purity fetishists

Huh? No new project of any note is using 2. Everyone has either upgraded to 3 or never even learned 2 to begin with.

Incorrect. There is tons of Python2 code being maintained, added to, and working diligently in production.

None of our projects had any trouble, rarely is an overstatement. Only if the project was huge and dealt heavily with string/bytes was it a problem. Everything else could be fixed mechanically.

Reply on news.ycombinator.com