Hacker News Reader

Comment by rsyring | Hacker News Reader

Can we please stop complaining about Python 2to3 already. Feel free to use and maintain Python 2 indefinitely or move to a different language. Most of the rest of us will move on since the conversions were not that hard and the improvements worth it and we were given 10+ years to do it.

Python is a fantastically productive language in many contexts. The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again. But whatever mistakes were made weren't malicious and they were learning as they went just like the rest of us. I greatly appreciate their efforts to fix the things that were broken, regardless of the hassle involved in upgrading.

View on HN

I refused to get involved in 2to3 migration projects for years.

I recently took care of a 2to3 migration involving a popular web framework and a few technologies for one of my employer's clients. The codebase is large and convoluted but I still found it surprisingly easier and way more interesting than I expected.

There is some actual difficulty in dealing with packages that have deprecated for years, and other gory details. But these days 2to3 drama is mostly a meme. The Python community, with a few exceptions, has done an amazing job at keeping things working.

I've done this for a few upstream projects.

If you want have a bug you want attention on in an upstream project, the easiest way to get it is to do a task for them first, and doing python 2 to 3 is a usually a pretty straightforward one.

I agree with what you say, but:

> The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again.

Saying that "probably do some things different" is quite an understatement.

Python 2 to 3 was an articulated train collision in slow motion that spans a decade. I'm sorry to say that, but it was monumentally awful.

I really, really wish that it didn't happen because I think it kept Python back at least 5 years.

People got really burnt by this upgrade process, and the answer was "we said we'd do backwards incompatible changes". "And what, are we going to have this every few years?" one might have though.

So, it isn't enough to tell people to "stop complaining about Python 2to3" and things might be done differently in the future or past.

> it kept Python back at least 5 years.

Yeah, it's done alright for itself hasn't it? Look at the stats, it's a top #2 language on Github, StackOverflow questions, Developer surveys or any other measure. Complaining that it could have achieved this 5 years earlier, I don't see the point. We can just appreciate where it's at now, and the people who were instrumental in getting us here.

> "And what, are we going to have this every few years?"

Isn't this the current situation now? Granted, not as big as Python 2 to 3, but there are backwards incompatible changes often.

What do you think Python would've been like had it never happened? I feel like Python is a success by any measure - especially considering it could've easily been replaced by javascript in a lot of cases without a whole lot of effort. I'm sure it kept it back to some extent, especially in the short term... but now it's thriving, and without some of the legacy stuff that would've turned off some people (e.g. handling of unicode strings).

The migration could have happened more smoothly if they had let both python2 and python3 code coexist in the same project. The problem was forking the language like they did. Why couldn't they make the same interpreter run both python3 and python2 code, for a transition period, deprecating python2 features gradually? So you could have had legacy python2 code and new code written in python3. The low level implementation is not that different to require that big break. The only major difference was on string types, but that could have been handled.

Breaking stuff as they did made no sense. And the problem is that after 10+ years that python3 came out in most Linux distribution if you type python you get python2, and it will probably ever will for maintaining compatibility with scripts that started with `#!/usr/bin/python` expecting it to be python2 and not 3. That is a problem for everyone, for maintainers of the distributions, for developer and sysadmins that cannot assume that the default version of python is 3 (you don't know how many time I wated debugging programs written by people that use Arch or whatever fancy distribution and assumed that /usr/bin/python is python3, and then the script breaks on the Debian production server...)

At least since they made a new language, they could invented a new extension for the files, so the `/usr/bin/python` executable could have detected the version to use based on the extension (launch .py3 files with python3, and .py files with python2).

They should've absolutely maintained a compatibility layer in Python3 for a couple versions (say until Python 3.5?) instead of splitting python2 and python3. Something similar to Perl's "use" feature flag.

The problem is, they are calling it a mistake but they were very well informed of the consequences of doing that they did. Everybody said "just fix integer division and print, finish the migration then start adding new features", and they went ahead and totally ignored that.

Whenever I work on a project that gathers interest in multiple places I get input like:

* If you do X you'll fail

* If you don't do X you'll fail

* X is neat but don't worry about it til later.

I imagine that the people in charge of python, a wildly popular programming language (at the time, and still), got all sorts of input, much of which conflicted. They chose a path and it was a bad choice, a mistake. This is clear in retrospect 13 years later, but at pycon in 2008 (a few months before python 3 officially released) the viewpoint was more mixed - a lot of people thought it was a good path and a lot of people thought it was terrible and others still thought the transition plan was annoying but would be cleared up pretty quick.

It's not so much the issue that it is clear that it was a bad decision. — they admit it, owe up to it, and I do not blame them for making the mistake and admitting it was unfortunate and that they learned their lessons.

What drives me to ire to no end are the many people that defend it on message boards with “Just upgrade; you had ten years.”, clearly not realizing the risks and money involved in such an endeavor.

It's as though the government suddenly start putting something in the water that improves it's health benefits, but will erode contemporary plumping pipes away in ten years and people say. “Just break down your walls and replace your plumbing; you have ten years.”. — it is quite expensive, and risks that my house fall over.

This is a reasonable take, but think about this from the perspective of engineers who've never made any mistakes in their lives. It's hard for them to fathom how anyone else can make a mistake like the Python devs did. The person you're answering clearly had the foresight to know what was the right thing to do, why can't everyone?

As an outsider: what was the problem with integer division?

Unless you come from a programming background where floor behavior is expected for integers, it was strange.

In Python 2: 20 / 15 = 1

In Python 3: 20 / 15 = 1.3333333333333333

To get the old behavior a new operator was added "//"

> Unless you come from a programming background where floor behavior is expected for integers, it was strange.

In all fairness, the C/C++ and Java communities were and are quite substantial.

Of course the rest of us ought to expect 4/3 as the result of 20/15 ;-)

Python’s separation between ints and floats is quite weak, so it didn’t make sense for `3 / 2` to give a different result from `3.0 / 2` (1 and 1.5 respectively). Python 3 changed the division operator to return 1.5 in both cases.

Python isn't strongly typed. You got an integer floor operation with -3 / 2 = -2, but -3.2 / 2 = -1.6. This led to errors when floats were expected but integers were possible among your inputs.

In Python 3, it always gives a floating-point result. You can use a floor division operator if you want to round everything (including integers) towards negative infinity.

See https://www.python.org/dev/peps/pep-0238/ for details.

Python is strongly typed, and your examples show the strong (dynamic) type system in action. An example of a language that is not strongly typed would be Javascript, where you can do 1 + "1". In Python, that gets you a typeerror, because Python is strongly typed.

Sorry, you're right, it's strongly typed but it is not statically typed.

In Python 2:

> 20 / 15 = 1

> 20.0 / 15 = 1.3333333333333333

> 20 / 15. = 1.3333333333333333

> 20. / 15. = 1.3333333333333333

In Python 3:

> X / Y is ALWAYS float division

> X // Y is ALWAYS integer division

This was managable in Python 2 but led to lots of float(argument) calls in every function that wanted to do float division. That lead to unnecessary calls, and unnecessary noise in the code.

The issue is that `int(x) / int(y)` would yield an int result in Python 2, but a float result in Python 3. So you had to find all the spots in your code where `/` was being used as an integer division operator and replace it with `//`.

Since Python 2 does not have any form of type annotation, this made the task hard to do automatically and error-prone.

Keep in mind that // was added to python in version 2.2a3 along with `from __future__ import division`. This gave everyone almost 7 years to fix up any existing code before the first release of python 3 even happened. 2.7 only got deprecated in 2020, giving developers a total of 18 years to fix their divisions.

I don't know what the PSF should have done to placate people. Keeping a core feature like division confusing like that isn't a good option, and extending support even longer seems ludicrous to me.

I guess they mean python2 1/2 == 0 python3 1/2 == 0.5 (threw me the first time when I moved as I use this to demonstrate the differences in integer devision in C based languages).

I started reading about Python long before I began programming in it (partially due to a deeply terrible book on the topic I purchased). I ran across the integer division bit and stopped cold. "That," I remember thinking, "is a language feature they will change."

Is that because you didn’t have experience with other languages? Because integer division in python2 worked in the way I had come to expect from Fortran, etc.

No, precisely the opposite: it is because I have tons of experience with many other languages.

I get why some people think a default integer division is a good idea. I do. Having said that, it was clear that Python was aiming itself at being a kind of glue language wherein various functions were obvious, and integer division is not obvious to someone picking up programming for the first time.

If I walk up to someone and ask "What's three divided by two?" I would expect more "One and a half" answers than "Two goes into three just the one time."

Eh, the Unicode situation was irredeemably awful in Python 2 because of the era it was of. Fixing that could never have fitted into what you describe. It had to be a significant breaking change, and even if Python 3’s string handling isn’t perfect (its strings are sequences of Unicode code points, not Unicode scalar values, so they can be ill-formed which was madness to allow; I say they should have gone all in on UTF-8 instead, which trades away some sorely misleading, tempting but practically quite useless functionality in favour of performance and memory usage), it’s vastly better than the situation in Python 2, where you pretty much just couldn’t be sure of anything—on many common configurations, you couldn’t even reliably output Unicode, and might not be able to at all without setting the PYTHONIOENCODING=utf-8 environment variable before starting Python, if my memory serves me correctly (I am glad to say that I haven’t had to deal with python2 for five years now).

True, though that's one of the things that 2to3 couldn't deal with, because it's not possible to do so automatically. It can't have had any idea what was supposed to remain a string, and what was supposed to become bytes.

My problem was that there were other unspecified things that 2to3 couldn't deal with. Imagine the world where the upgrade guide was to run 2to3 and then it had explanations of how to address the remaining string issues. Instead, I found the attitude of "run 2to3, it'll probably work but if it doesn't, then you'll have to figure it out yourself."

In my opinion, the issue was that they used this feature as an excuse to have other breaking features that weren't really needed. I think besides the string upgrade, every other breaking feature should have worked with 2 to 3.

Then they could have had a clear upgrade guide that is to run 2 to 3, then clear explanations on how to fix all the remaining string issues. Instead, I have no clue what 2 to 3 fixes and what it doesn't, leading to "2 to 3 will probably work, but if it doesn't, then you'll just have to figure it out yourself."

That's unacceptable, and as a result caused many libraries to take a long time to upgrade. Personally, I had to spend 10 YEARS to wait to upgrade to Python 3 due to the wxPython library that all of my projects used that took that long to upgrade to Python 3.

yeah, I forgot to include unicode but I actually think they should have completed the print() and integer division change migration (without a major version bump, with import futures) and then done a major version bump (to 3) specifically for Unicode. And I agree- utf-8 has clearly established itself as the preferred encoding.

> Most of the rest of us will move on since the conversions were not that hard and the improvements worth it and we were given 10+ years to do it.

You are describing it as if it is for everyone easy to convert.

That might be true for fast-paced start-ups and web companies which rewrite their code every few years.

I work in science, in projects which have complex infrastructure and are developed in the course of many years. There is a vast amount of code which is written in the language and virtually nobody has time to rewrite and convert it. Me neither. As a consequence, Python is not really ideal for this kind of work, and I will tend to avoid it in my next job.

It might not seem to matter to you, but a good part of Python's success is actually because of things like Numpy, Scientific Python, and statistics and science libaries which were written by scientists. Some, like Konrad Hinsen (one of the authors of Numpy) are now even considering to move to another language:

http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stabilit...

The idea that converting is easy is also not generally true for any kind of infrastructure. Consider the Python Wiki:

https://wiki.python.org/moin/

At the bottom of the page, you find the note "MoinMoin powered". The Wiki software is the MoinMoin wiki software. It is the only large wiki software with multi-markup support written in Python. It has so far no Python 3 port. This is because people have written it in their spare time, and they have no time to rewrite it. Do you have the time? Would you volunteer for that?

From the point of a naïve language author or a library author, change is part of life and users of their software should just upgrade to the latest version, as you write. But this is not friendly to the users of the software. Users really have other things to do than to constantly rewrite their software according to your latest syntax change or broken dependency.

More proficient languages and infrastructure projects, like C, C++, Common Lisp, Java, and the Linux Kernel, do not have this approach - they maintain strict backwards compatibility. "WE DO NOT BREAK USER SPACE", in all caps - do you recognize that phrase? You can google for it. And this rule has a reason. You can think a few minutes how successful that project would have been without this rule. What if if you want to run a new Python on Linux, you first need to upgrade your Linux kernel, and then you need to upgrade your Linux distribution, and then you find that the new distribution release does not support HTML1, so the library you were going to use is broken, and, oh, your printer won't work with it? This is more or less what Python expects from its users.

The Python developers do not adhere to this - they break backwards compatibility. If one reads the fine print about the release cycle, it says clearly that there might be breaking changes even in minor releases.

At the same time, the Python developers did not want people to realize that Python 3 is not the same language, and used the same file extension for the new language. This has as an effect more backwards incompatibility, for example there are libraries which continue to support Python 2 for some time and then stop that support - without using a major version number increment. As a consequence, Python 2 projects which use such a library will break if using normal tools, which assume that semantic versioning is adhered to. In other words, one cannot rely on that Python library authors adhere to semantic versioning.

All this is, regardless how much Python developers fuss about it, a choice. It would have been possible to keep Python 3 backwards compatible without breaking anything. Common Lisp has done it and it is decades older. But this has not happened and as a consequence, Python is less suitable for some types of projects and infrastructure.

That's not something to complain about, and I don't. It is just a fact.

> Can we please stop complaining about Python 2to3 already. Feel free to use and maintain Python 2 indefinitely or move to a different language. Most of the rest of us will move on since the conversions were not that hard and the improvements worth it and we were given 10+ years to do it.

Have you done such an upgrade with a million line codebase?

No company wants to do that with such a large codebase with the potential of regressions that can cost quite a bit of money.

Reading this article makes it sound as though the experience taught him a valuable lesson and that he agrees that in retrospect Python 3 was a mistake, and it was.

There is quite a good reason why so many platforms and languages take indefinite backwards compatibility quit seriously. — it is simply a very demanding thing of large codebases to rewrite to a newer version of the language with all the potential of subtle regressions.

> The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again.

The article suggests that if they could go back in time, they would not do it and make no backwards incompatible change to Python 2.

> But whatever mistakes were made weren't malicious and they were learning as they went just like the rest of us. I greatly appreciate their efforts to fix the things that were broken, regardless of the hassle involved in upgrading.

It was certainly not malicious, but it was indeed a bad idea which is what the complaining is: that Python 3, or any such backwards incompatible changes should have never happened.

This is why Rust uses editions and many other languages a system where backwards incompatible changes must explicitly be imported and the old behavior continues to be supported indefinitely.

> Have you done such an upgrade with a million line codebase?

Yes. With a transition plan everything was ratcheted over to python3 syntax, then test compatibility. Now developers no longer have to remember the nuance between python 2 and 3 and how to write compatibility layers for both.

Now we no longer need to worry about maintaining or working around issues in dependencies that haven't been updated in a decade, and we don't need to teach python 2.7 to new developers.

> Yes. With a transition plan everything was ratcheted over to python3 syntax, then test compatibility. Now developers no longer have to remember the nuance between python 2 and 3 and how to write compatibility layers for both.

Then tell me, what were the costs of the transition when completed? and what where the projected costs when starting?

> Now we no longer need to worry about maintaining or working around issues in dependencies that haven't been updated in a decade, and we don't need to teach python 2.7 to new developers.

You wouldn't if Python 3 didn't exist either.

> It was certainly not malicious, but it was indeed a bad idea which is what the complaining is: that Python 3, or any such backwards incompatible changes should have never happened.

> You wouldn't if Python 3 didn't exist either.

This is a bit of a loony argument. There are so many useful features post py27 that may not have been possible without the breaking changes 2to3 introduced. Especially the horrific bytes/str situation in 2.7.

So your argument is that Py3 shouldn't have happened, despite all of the people actually contributing to the language wanting it so (I wonder how much does company with your multi-million line project contributes to cPython's development?) and it unlocking future improvements that have driven Python adoption and usage into the stratosphere... because your company didn't want to invest any time or resources for ongoing maintenance? Of which upgrades are an obvious and important part?

> This is a bit of a loony argument. There are so many useful features post py27 that may not have been possible without the breaking changes 2to3 introduced. Especially the horrific bytes/str situation in 2.7.

This situation could easily have been solved without breaking changes in a multitude of ways.

> So your argument is that Py3 shouldn't have happened

Yes, my argument is absolutely that none of the changes warranted breaking backwards compatibility.

> despite all of the people actually contributing to the language wanting it so (I wonder how much does company with your multi-million line project contributes to cPython's development?) and it unlocking future improvements that have driven Python adoption and usage into the stratosphere... because your company didn't want to invest any time or resources for ongoing maintenance? Of which upgrades are an obvious and important part?

That's merely an ad hominem, not an argument as to why you believe that the changes merited breaking backwards compatibility when there were far less intrusive ways to realize solutions that were close to as convenient but did not break backwards compatibility.

> Yes, my argument is absolutely that none of the changes warranted breaking backwards compatibility.

> That's merely an ad hominem, not an argument as to why you believe that the changes merited breaking backwards compatibility when there were far less intrusive ways to realize solutions that were close to as convenient but did not break backwards compatibility.

You haven't made an argument, nor given any alternatives. You've just complained that the free work of others wasn't to your specifications.

How would you propose handling the str/bytes situation without breaking backwards compatibility?

Refusing to break backward compatibility is how you end up with some of the uncomfortable anachronisms that can be found in PHP and MySQL and other things. A language that did backward compat arguably well is Java but it had to be extremely conservative, a property that has made it popular to replace wholesale with languages that are nothing like Java.

If it helps, you can convince yourself that Py3 is a different language that just happens to be very similar to Py2, and that Py2 needs new maintainers. This happens all the time.

> This situation could easily have been solved without breaking changes in a multitude of ways.

Please, do elaborate. (I'll note that if I were redesigning python3, I wouldn't settle on the design for unicode that py3 uses, something more rust-like is probably a bit better, but there's no way to transition from string-literals-are-bytes to string-literals-are-unicode without breaking backwards compatibility)

> You wouldn't if Python 3 didn't exist either.

It does exist. Python 2 is dead. The transition was more painful than it perhaps needed to be, but is IMO a better language for it than 2.7, or 2.6.

PEP-3000 was 15 years ago.

Complaining for decades about mistakes, that they admit, with hindsight, won't change anything.

> and that he agrees that in retrospect Python 3 was a mistake, and it was.

You need to read it again. What he actually said was he should have provided more support to developers migrating code, not that he wouldn’t have done it.

> Feel free to use and maintain Python 2 indefinitely or move to a different language. Most of the rest of us will move on since the conversions were not that hard and the improvements worth it and we were given 10+ years to do it.

Maintaining Python 2 indefinitely is an option actively being taken away from people. More importantly, fuck anyone who wants to direct my budget! There's things written in Python 2 ages ago that don't justify additional developer time in my company. Except the community is making us do it.

If you think migrating some ancient Django app to modern Django is trivially easy, why don't you come do it for us?

> Maintaining Python 2 indefinitely is an option actively being taken away from people

Where do you see this happening? All I see are people and organizations deciding that they don't want to do the work of maintaining Python 2.

Python 2 removed from homebrew.

Python 2.7 is not getting any security/crypto updates backported. Eventually PyPI's SSL configuration will change to a point where Python 2.7 won't handle it and then that's it. Game over, no more pip for you, even if you froze on < 21.0.

There are Linux distros out there in LTS until late into this decade that shipped with Python 2.7. Ubuntu 18 comes to mind, which EOLs in 2028. They've got a real hard problem to solve.

Python 2.7 is not getting any security/crypto updates backported.

There are several projects out there that are supporting python 2.7 with security updates. Tauthon is probably the biggest one.

Thank you. Yes it is good to see some projects stepping up the fill the gaps. I only wonder as time goes on how well socialized these will be...and if people who suddenly find themselves in the position of maintaining an old Python code base will be able to find out what to do.

Tauthon is kind of interesting because in addition to guaranteeing compatibility with the latest official 2.7 release they are also trying to back-port as many python 3 feature as makes sense without breaking the backwards comparability.

Oh, you mean other people maintaining Python 2 for you is being taken away.

You can maintain Python 2 yourself just fine, though, if you like.

heh

You can even still get someone else to maintain it for you! Just not the Python Software Foundation. If it's an important piece of your business it may make sense to have a support contract in place regardless.

How does that "actively take away from people the option to maintain Python 2 indefinitely"?

It appears to only take away from people the option to use the unmaintained Python 2.

But if the libraries and extensions are not maintaining compatibility, you need to freeze your dependencies, and sometimes that means freezing the dependencies of your dependencies, or even your OS, and then you hope your machine doesn't break.

It's not a tragedy, backwards-compatibility is a burden, and I understand maintainers, especially volunteers not wanting to do it. But migration is a pain, I found the conversion was not too hard, but testing the migrated scripts for breakage was time consuming.

FYI, the cause of my latest migration was that the latest version for the mysql-connector for python no longer works with 2.7. See the changelog entry for v8.0.24 here: https://github.com/mysql/mysql-connector-python/blob/master/...

So something to consider if you want to stay on Python 2 forever.

>More importantly, fuck anyone who wants to direct my budget!

Is your budget going at all to maintaining Python 2 or the ancient Django you use? I'm gonna guess not.

If my local electronics shop can still operate fine on win3.1 I'd imagine that there's a fair amount of places where no maintenance at all will be needed

If anything, this is the normal state of things in business.

We all know about how COBOL is alive and kicking but Python's core team decided that reality doesn't apply to them.

Sure, IBM will gladly sell you an up-to-date COBOL. Just like someone will gladly sell you an up-to-date Python2. It ain't free.

Nobody forced the upgrade to COBOL and it still runs just like you can still run whatever old version of Python you're tied to, with the same downsides as if you ran an ancient version of COBOL.

No that's the point. It's something that doesn't make sense to spend any more money on. It's a tool that's being replaced but that process takes years. Unfortunately not soon enough before the end of Python 2.

Python 2 isn't going to just stop working. You just have to accept the downsides if you're not willing to budget for migrating.

How much are you paying for Python again?

You chose to wait this long. You could have spent a little budget to maintain your software but you didn't and now you're paying the price.

Reply on news.ycombinator.com