Python 2.7.18, the last release of Python 2 | Hacker News Reader

Python 2.7.18, the last release of Python 2

pythoninsider.blogspot.com

339 pointsvishnu_ks338 comments

For those with Python 2 baselines that aren't (or will never be) ported, PyPy will support version 2 basically forever [0]. PyPy supports a huge number (but not all) of Python packages, including numpy [1]. Moreover, PyPy is significantly faster in many cases [2], and for the numerical types of things I like to write [3], it's amazingly faster.

    > time pypy mandel.py > pypy.pgm
    seconds: 0.318101

    real        0m0.426s
    user        0m0.396s
    sys         0m0.013s

    > time python2 mandel.py > python2.pgm
    seconds: 30.141954

    real        0m30.156s
    user        0m30.136s
    sys         0m0.003s

That's just a silly Mandelbrot example, but for numerical algorithms like this PyPy is nearly 100 times faster than Python2, and that includes startup cost for the JIT!

I'm not in any way associated with the PyPy project, but I can't help but believe a sane world would've moved from Python to PyPy in the same way everything moved from Brandon Eich's original JavaScript interpreter to the modern JIT ones like V8.

[0] https://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-s...

[1] http://packages.pypy.org/

[2] https://speed.pypy.org/

[3] https://godbolt.org/z/J9Xwp6

I can't help but believe a sane world would've moved from Python to PyPy

Yes, and this is my fundamental complaint with the Python 3 transition. It took probably millions of engineer-hours, and the result was better Unicode support and a handful of minor features most of which could have been added to Python 2 just as easily. I suspect most users would gladly have traded those benefits for a 10x performance improvement.

> millions of engineer-hours, and the result was better Unicode support and a handful of minor features

Yes, and I wish the better Unicode support had been implemented similarly to how Go does it - one string type, and you use it to hold UTF-8 if needed. In other words, they could've simply deprecated the Python 2.X unicode object and added libraries to extract code points, or grapheme clusters, or perform normalization etc... This seems much simpler and more "Pythonic".

I guess everything is 20/20 in hindsight.

I totally agree with you. I hope we still have a chance to do it well with PyPy.

> and the result was better Unicode support

Different Unicode support. And worse bytes support.

What could previously be done using python -c "..." is now long, horrible and ugly.

> Different Unicode support. And worse bytes support.

I feel like you're the first person I've seen on the planet to echo my sentiments on these. I expect a lot of people will jump here to tell you you're wrong like they have to me, so just wanted to let you know I've felt exactly these pains and agree with you.

I am hoping this is in agreement. py2’s flexibility to handle utf8 bytes without fuss is amazing. Then people come up with all kind of purity reasons to make it more complicated.

Take out "utf8" and I'll agree ;)

The fundamental problem as I see it is that "string" is a grossly leaky and misunderstood abstraction. The string type is not the same thing as a "text" type. It's being used in all the wrong places for that purpose. People treat "string" like it means "text", but in so many places where we deal with them, they just aren't (and should never be) text. Everything from stdio to argv to file paths to environment variables to "text" files to basically any interface with the outside world needs to be dealt with in bytes rather than text if you care about actually producing correct code that doesn't lose, corrupt, or otherwise choke on data.

C++ understood this and got it right, preferring to focus on optimizing rather than constraining the string type. Many other languages did pretty well by avoiding enforcing encodings on strings, too. And Python 2 defaulted to bytes as well, and only really cared about encoding/decoding at I/O boundaries where it thought it can assume it's dealing with text (though it sometimes didn't behave well there, and yes it got painful as a result). Then Python 3 came along and just made everyone start treating most data as if they're inherently (Unicode) text by default, when they really had no such constraints to begin with.

It boggles my mind that Python 3 folks like to beat the drum on how Python 3 got the bytes/unicode right without taking a single moment to even notice that most strings people deal with aren't (and never were!) actually guaranteed to be in a specific, known textual encoding a priori. They were just arrays of code units with few restrictions on them, and if you want to write correct code, you're going to have to deal with bytes by default (or something else with similar flexibility) instead of text. It would've been totally fine to introduce a text type, but it fundamentally can't take the place of a blob type, which is the language of the outside world.

"The outside world", by and large, also speaks Unicode.

Java uses UTF-16 throughout, including file paths. So does .NET. All Apple platforms are UTF-16. C++ - if you just look at stdlib, sure, it's byte-centric; but then look at popular frameworks such as Qt.

In practice, this means that, yeah, you can have that odd filename that is technically not Unicode. But the vast majority of code running on the most popular desktop and mobile platforms is going to handle it in a way that expects it to be Unicode. Why should Python go against the trend, and make life more complicated for developers using it in the process?

> py2’s flexibility to handle utf8 bytes without fuss is amazing

Without fuzz? No, sorry, it was anything but.

First of all it would default encoding to "ASCII". Have any whiff of non-explicitly handled UTF-8 and it would just go bang at the worse time possible.

That was a stupid decision

"Oh but there was setdefaultencoding" Yeah here's the first result for that https://stackoverflow.com/questions/3828723/why-should-we-no...

So no, Python2 way of dealing with Unicode was the most annoying way possible, because hey who needs anything but ASCII right?

> Python2 way of dealing with Unicode was the most annoying way possible

The part about defaulting to ASCII is annoying, yes. And using sys.setdefaultencoding to change the default would still be annoying, yes. The reason for that is that any default encoding will be annoying whenever the actual encoding when the program is running doesn't match the default.

The correct way to fix this problem is to not have a default encoding at all. Don't try to auto-detect encodings; don't try to guess encodings. Force every encode and decode operation to explicitly specify an encoding. That way the issue of what the encoding is, how to detect it, etc., is handled in the right place--in the code of the particular application that needs to use Unicode. It should not be handled in a language runtime or a standard library, precisely because there is no way for a language runtime or a library to properly deal with all use cases of all applications.

What Python 3 did, instead, was to change the rules of default encodings and auto-detection/guessing of encodings, so that they were nicer to some use cases, and even more annoying than before to others.

I agree that it was easy to shoot yourself in the foot, and if you did have to deal with unicode, it was often a pain, but at the same time, Python's simplicity and ease of use is what makes it great, with the ability to do something cleaner if you choose to. You can choose to type annotate all your code and make it better. You can choose to organize your code in however package structure you want or keep it all in one file. The language doesn't force any of that onto you. That's the approach I think would've been much nicer and Pythonic in my mind. Now you're forced to use a much clunkier bytes/str paradigm, which yes will make your life much nicer in the 10% of the time when you'll need it, but the other 90% of the time will just be slightly more annoying. Similarly, I may be alone in this, but having to put parens around print statements is also annoying 99% of the time, but nice that 1% of the time I need to pass it as a function or pass it some extra arguments.

> Have any whiff of non-explicitly handled UTF-8 and it would just go bang at the worse time possible.

Whereas python3 is just waiting to explode the moment there are bytes in your UTF-8 that are invalid.

Oh the http request got truncated to leave invalid utf-8 in an otherwise fine utf-8 response? FUCKING ERROR.

Except for that part where it would happily implicitly convert them to/from a Unicode string in any context where one was needed or present... using ASCII, rather than UTF-8, as the encoding.

> I feel like you're the first person I've seen on the planet to echo my sentiments on these.

There have been plenty of people with similar sentiments. I'm one of them. I have felt ever since I first looked at Python 3 that the ways in which it broke backward incompatibility were heavily skewed towards a few particular use cases and did not take into account the needs of all of the Python community.

Got any examples?

I didn't want to reply to the PyPy comment and be negative, but I haven't really gotten speedup from the few times I've tried PyPy. In fact I've generally gotten slowdowns. Definitely not 10x improvements. I'm not sure what the reason might be though, because the rest of the world seems to think differently.

It's been a while since I used PyPy, but JITs in general warm up over time. If you have hot loops with heavy arithmetic and no branches, that's usually best case scenario for JITs. If you have branchy non-uniform control flow, that's the worse scenario. So it really depends on your usage - you may be paying the JIT overhead costs with little benefit.

Spot on, and the difference between a tracing JIT vs a method JIT can be night and day too.

Really depends on the kind of work you're doing. As mentioned above, numerical and looping code will see the most benefit. I used it heavily when doing Project Euler problems, and those would easily see 10-100x speedups, taking some problems from 5-10m run time to seconds.

Yup. I like Python3, it's better than 2, and has lots of good new features, but so many of them could simply have been added onto 2 without requiring this huge painstaking migration that cost an unbelievable amount of effort worldwide.

Contrast with Java, which has made much more substantial changes to the language over all these years, but goes to great pains to support backwards compatibility. Upgrading major Java versions was never remotely as painful, and as a result, huge amounts of engineering time was saved.

Python 3 features in Python 2 was entire purpose of Python 2.7. All features it came with were backports of python 3 features. They could continue doing that, but then we would never fix the Unicode.

There's so much whining, when they gave over 5 years to do the migration. I think the whole problem was that they gave too much time, and people were thinking it will be like that forever.

Also, Java is not a dynamic language, the type system allows for easy fix of issues like these, the python solution to do that is mypy, but it requires work by adding types.

> There's so much whining, when they gave over 5 years to do the migration.

You make it sound so generous of the Python maintainers to "give over five years" before breaking backwards compatibility--something that C, Java, JavaScript, C++, and probably most other languages haven't done for decades, or in some cases ever. Programming languages are serious basic infrastructure, programmer time is valuable, and Python's installed base is large enough that many man-centuries were probably wasted on this migration.

If the Python maintainers had not given as much time as they did, I doubt the Python community or ecosystem would have transitioned any more quickly than they did. More likely would have been some fork or alternate implementation of Python 2 becoming the new standard.

Okay, but how many man-centuries have we spent fixing buffer overflows in C because the type system can't check that without breaking reverse compatibility?

Reverse compatibility isn't free.

It’s not like C is the only programming language to have never broken backwards compatibility, or that conflating byte arrays and strings is as dangerous as buffer overflow.

People have made memory-safe compilers for ANSI C (e.g. Fail-Safe C.)

Unfortunately clang and gcc don't (yet?) have --safe or an x86-64-safe compilation target.

> better Unicode support

I guess it's a matter of emphasis, but I'd say it has different Unicode support. It's better mildly for some use cases, but worse for others.

It's bloody horrible for one use case in particular: when you know the text is readable and mostly ASCII based and you are only interested in the ASCII bits, but don't know the encoding. That is the position you find yourself in for any designed in pre-unicode times, and that happens to include just about every file in a Unix file system.

The solution in those circumstances is to treat everything as bytes (b''). That wasn't even possible in the beginning. Now it mostly works, but all with hundreds of corner exceptions (like Exception messages, so you can't easily include a Unix filename in an error message).

It's only 10x for web. Anyone using python for batch processing won't see the gains because the work is done before pypy warms up.

But yes the only people who wanted better unicode were web people. So they could have been better served moving to pypy.

Pypy is great, but numpy is even faster (2-3x over your pypy example), and the code is less strange. Here's the mandel function in numpy:

    def mandel(data: np.ndarray):
      c = data
      z = c
      for j in range(255):
          z = z**2 + c

(from here: https://scipy-lectures.org/intro/numpy/auto_examples/plot_ma...)

Yeah, that's the thing with silly examples (like my Mandelbrot program). For real code, I frequently need to write low level loops which aren't easily expressed as parallel operations. If numpy doesn't have it, or if you can't figure out how to parallelize it, you're screwed.

Moreover, for some very common things in signal processing, like working with evens/odds or left/right parts of an array, the parallel numpy operation will create lots of temporary arrays and copies.

And for what it's worth, your version of mandel should work with PyPy. So you can have your cake and eat it too.

EDIT: I should add the reason my code is "strange" is because I wrote it so I could do a one-to-one comparison with other languages which don't have builtin complex numbers. Maybe I should've cleaned that up before posting.

> Moreover, for some very common things in signal processing, like working with evens/odds or left/right parts of an array, the parallel numpy operation will create lots of temporary arrays and copies.

iiuc, this shouldn't be correct. given an ndarray A, `A[::2, 1::2]` will provide a (no-copy) view of the even rows/columns of A. Same with A[:len(A)/2] to get only half of A.

> And for what it's worth, your version of mandel should work with PyPy. So you can have your cake and eat it too.

Indeed, most of the scipy stack works with Pypy, it's great.

As soon as you do any operations (addition, subtraction, etc...) on those views, you're going to get temporary arrays.

For instance, your Mandelbrot example doesn't even use views and it creates two temporaries the size of the entire array on each iteration:

    for j in range(255):
        t = z**2    # create a new squared array
        u = t + c   # create a new summed array
        z = u       # replace the old array

And all of this is ignoring how inconsistent numpy is about when it creates a view and when it creates a copy.

Unnecessary temporary arrays is definitely a major source of inefficiency when working with NumPy, but recent versions of NumPy go to heroic lengths (via Python reference counting) to avoid doing so in many cases: https://github.com/numpy/numpy/blob/v1.18.3/numpy/core/src/m...

So in this case, NumPy would actually only make one temporary copy, effectively translating the loop into the following:

    for j in range(255):
        u = z**2   # create a new squared array
        u += c     # add in-place 
        z = u      # replace the old array

Your general point is correct, although in this specific instance, replacing the loop body with

    z **= 2 
    z +=c

gets rid of the temps. But yes there are cases where that isn't possible.

This gets rid of temporary arrays, but this still isn't optimal if z is large. Memory locality means it's faster to apply a scalar operation like z2+c in a single pass, rather than in two separate passes.

Explicitly unrolling loopy code (e.g., in pypy or Numba) is one easy way to achieve this, but you have to write more code.

Julia has some really nice syntax that lets you write things in this clean vectorized way but still get efficient code: https://julialang.org/blog/2017/01/moredots/

Vectorization FTW

The python is the least of issues, you can just keep 2.7.18 and 10 years from now it will still work. The problem is dependencies. Developers of packages couldn't wait until they can drop all the cruft that python 2.7 was required.

https://python3statement.org/

Just do yourself a favor and migrate your codebase, Py3 is much more enjoyable to program in. I feel like all those vocal Python 2 supporters never had a chance to write a new code in Python 3. If you only did 2->3 migration you might hate Python 3, because it doesn't let you run a broken code that Python 2 happily executed, but if you can write Python 3 app from scratch you don't even notice the unicode, text is just text.

"A broken code that happily executed"?

I think that's unfair. There was plenty of code that was out there for 10 years or more that was working completely fine and had to be ported. One of the most frustrating things I had to do was completely rearchitect some legacy binary file reading/writing because of the changes to how Python handled bytes. That code was out there as open source in the wild, stable, and was being widely used, and it basically required a full rewrite underneath the API.

One of the most frustrating things was that many packages we used as dependencies took ~5 years to port to Python 3, and then dropped Python 2 support immediately, leaving us with no choice but to use the old version for some time. We'd done a lot of the easy stuff already (2to3 on all files), but lots of the non-trivial things were the interactions with other packages so couldn't be touched until they had themselves got a Python 3 version.

> many packages we used as dependencies took ~5 years to port to Python 3, and then dropped Python 2 support immediately

A lot of packages were held back by early criticism of the move and the extended timeframe allowed to 2.x.

If 10% of the stop-energy dedicated to shit on 3 would have been put into supporting the effort, things would have gone differently. But most people dragged their feet and this is the result.

I saw pretty great results switching a pretty “standard” high traffic Python web app (Flask & uwsgi) over to run on PyPy. We saw about 30% faster response time for HTTP requests, several times the number of tasks/second on the workers, and we were able to scale down the total instances needed. Mostly the typical Python web app libraries just worked, and I spent a couple of days making sure everything was stable, but it was a great success overall.

This is pretty interesting.

Years ago I had a bunch of code that was basically just matrix multiplication with some large-ish matrices, and then taking some eigenvectors/eigenvalues at the end. At the time I found the same thing -- if I decomposed things into simple lists of numbers for vectors and lists of lists PyPy was way faster.

I just had the opportunity to brush this code off in Python 3 and run it as-is, and it performs much better than it used to. But I am always curious to hear about these cases.

PyPy really is a wonderful project.

Came here just to say that PyPy is amazing, from my somewhat limited exposure, it fulfilled my use case very well with a speed gain.

I am interested and would like to learn more. So do you just

    sudo apt install pypy3

and then

    pypy3 -m pytest /my/python/app

and if things go well you either got a 5-10x speed up or an insufficient test suite?

Another casualty of python 3?

Honestly, I think Python's C API exposing so much of the internals of the implementation is the real problem. You can basically see every pointer in every struct, including tons of things you shouldn't need. Large packages inevitably end up using some unfortunate detail which couples them tightly to CPython, and this makes using those packages with PyPy nearly impossible. The fact that PyPy got so many of those to work as well as they did (numpy stands out) is a testament to their talent and stamina.

I believe this is also a huge part of the reason why migrating from CPython version 2 to version 3 was delayed. I've adapted a few small C extension modules to run under both, and using #ifdef for the special cases to support both was unpleasant. So I imagine that any large package which needed to support both through the transition really suffered for it.

Python 3 has "limited API", which is much better in that regard.

https://www.python.org/dev/peps/pep-0384/

Python is currently one of the most successful languages out there. Most of the growth at this point is coming from python 3. So, from that point of view, it's been a huge success.

If at this point you are still stuck on 2.7; you probably don't care a lot about updates in any case; including point releases. It's been well over a decade since it was made clear that this was going to end. So, IMHO the impact to remaining 2.7 users is minimal. They were in any case extremely conservative updating and are probably also running lots of other outdated stuff like Red Hat / Ubuntu versions that long dropped out of LTS, etc. That's fine and valid but at this point you shouldn't be surprised that you are on your own. If you didn't plan for this, that's on you.

From a security point of view that just means you probably don't want to run unprotected 2.7 servers running e.g. a web server. But otherwise it's fine if you shield it a bit. Lots of python is more about other types of jobs where the impact of security vulnerabilities is much less.

And, I'm sure that if there's demand, somebody might actually step up to do the occasional patch release if it is really needed. This has also happened in the Java world where several companies provide support for openjdk 6, 7, and 8 where Oracle no longer supports that (v8 stopped getting public updates already; you can still pay for some extended support but that too is being ramped down). I imagine, e.g. Red Hat might step up here as they seem to have continued to ship this for quite long and their LTS cycles might out run the python 2.7 cut off date.

This isn't a knock per Perl, but looking back, it's really interesting to see how the two communities handled their respective transitions: Python 2 to Python 3, and Perl 5 to Perl 6 (now called raku [1])

I say it isn't a knock because I think they were equally fine with the goals: Perl was looking to make a bold break towards an unknown future [2], and Python wanted a very slow and sustainable migration.

I'm glad to see Python 3 go mainstream, I'm glad that Python 2 succeeded so well, and I'm glad there are segments of computer science that still throw mugs and aim for the moon.

[1] http://blogs.perl.org/users/ovid/2019/10/larry-has-approved-...

[2] https://www.nntp.perl.org/group/perl.packrats/2002/07/msg3.h...

You have it backwards. Perl 6/Raku is a completely new language, the mistake there was to call it "Perl". Perl5 on the other hand has handled its evolution much more gently than Python 2 -> 3 did.

There's more to it than that. Looking at its early history [1], it is clear that Perl6 was conceived/intended to be the next version of Perl after Perl5 e.g.

First, Perl will support multiple syntaxes that map onto a single semantic model. Second, that single semantic model will in turn map to multiple platforms.

Multiple syntaxes sound like an evil thing, but they're really necessary for the evolution of the language. To some extent we already have a multi-syntax model in Perl 5; every time you use a pragma or module, you are warping the language you're using. As long as it's clear from the declarations at the top of the module which version of the language you're using, this causes little problem.

There were even plans for a translator [2] similar to Python's 2to3 tool

Larry Wall and others are already working on a Perl 5 to Perl 6 translator, which will be able to translate (most) Perl 5 source code to the equivalent Perl 6 syntax.

In addition, Perl 6 will provide a "Perl 5 compatibility mode", allowing the compiler to directly execute any code that it recognizes as being written in Perl 5.

[1] https://raku.org/archive/doc/design/apo/A01.html

[2] https://raku.org/archive/faq.html

Those were the plans. In the meantime, Perl 6 has been renamed to Raku (https://raku.org using the #rakulang tag on social media).

Integrating Perl code in Raku can be done with the excellent Inline::Perl5 module (https://modules.raku.org/dist/Inline::Perl5:cpan:NINE). In fact, that efficiency of that module basically killed the "parse Perl code in Raku" project.

Raku isn't a completely new language. It's very clearly a descendant of the Perl lineage.

Yeah but I think they have a point. As a former Perl user and current Python convert I do think the mistake was to call Perl 6 Perl.

It gave me at least a false sense of thinking Perl 5 was done and going to be replaced.

At the same time I found Python to be much easier to write, maintain and I became attached to the structure that PEP provided.

There were a lot of other factors that made me switch, but that particular point of not calling Perl 6 Perl made me think.

If you're interested in what actually changed in this update, the release notes are here: https://github.com/python/cpython/blob/2.7/Misc/NEWS.d/2.7.1...

Still know a heck of a lot of people using it. Even know one researcher using it for a new project.

It will be decades before the final Python 2 program goes offline.

Ya, there's no way this is the "final" release. Maybe by the core Python team, but it will be forked to fix bugs. Ten years from now there will still be Python 2 code running critical infrastructure at various companies, and the most responsible path to address discovered issues in the runtime will not be "rewrite the application to work in Python 3!" but "upgrade the interpreter to this community-vetted fork of 2.7.18".

Mumble mumble something about conflating languages with implementations.

What's use of Python 2 if you can't use libraries[1]?

It will only get more difficult to maintain your app.

[1] https://python3statement.org/ - note many libraries weren't even waiting until 2020. It is a lot of work to maintain code with python 2 cruft. Not all packages are listed there, for example Django is Python 3 only, starting from 2.0 (currently at 3.0)

> What's use of Python 2 if you can't use libraries[1]?

Unless some python 3 fanatic goes out of his way to write a python 2 library deleting virus the existing code wont disappear. Also some of these pledges only limit feature releases, afaik numpy planned to still provide a long term support version with bugfixes for python 2. It also helps that python already comes with a lot of build in bells and whistles so third party libraries aren't always necessary either.

Sorry I wasn't clear nothing happens if your application doesn't change, but if you do sooner or later you'll be forced to upgrade your dependencies (could be a bug that you just found, maybe a bug, or maybe a performance improvement) if the updated version won't work on your python it will be tough. You'll have choice to either migrate your app to python 3 or fork the library and backport fixes.

You might be lucky and someone else might do that for you, but it will be harder and harder with time. Already according to JetBrains survey in 2019 (I believe) about 80% of people surveyed they already use python 3.

As for numpy I just checked[1] and the only wheels they are providing for the latest version are 3.5+ the package also says that it is 3 only.

[1] https://pypi.org/project/numpy/#files

The problem isn't that Python 2 is bad. Python 2 is a fantastic language. The problem is that the maintainers of Python decided to break backwards compatibility and force library developers to support what are essentially two different programming languages.

I don’t think there was any clear path around that, though. The single biggest change was that Python 2 pretended that text and binary data were the same datatype, where Python 3 correctly makes you distinguish between the two. There’s not really a great way to roll out that major change without breaking tons of stuff along the way. And, well, if you’re already making a backward-incompatible version, here’s this checklist of other breaking changes you might as well bring along for the ride.

And that raises an obvious question: why didn’t every other programming language immediately break backwards compatibility when UTF-8 became a de facto standard?

> And, well, if you’re already making a backward-incompatible version, here’s this checklist of other breaking changes you might as well bring along for the ride.

Sorry, that doesn’t track. Treating quoted strings as UTF-8 by default instead of ASCII-or-arbitrary-bytes would have been a small migration that would not have taken over a decade to complete.

some would say that the distinction was in the wrong places, like assuming that the command line arguments or file paths were utf8

> correctly

Sometimes it’s better to be correct and also yield to the common good.

You can keep using the libraries you're already using. That's totally fine for many applications.

More likely by then the push to "RIIR"™ will be so overwhelming that most people won't be able to help themselves.

Glad to have struck a nerve with the Rust evangelists.

> Mumble mumble something about conflating languages with implementations.

So your claim is that "Python 2" is a language spec, not an implementation? And that there will be future releases of this language spec in the future? I doubt it.

I agree it's likely that there will be people wasting their time maintaining an interpreter fork, but that will not be Python-the-language (a trademarked term BTW), it will be a fork of the implementation.

No, my claim is that, while ceasing development of the language Python 2 is wholly sensible, ceasing development of the implementation Python 2 (CPython specifically) is not (due the almost certain existence of latent bugs). My "mumble" at the end was meant exactly in reference to that.

I suppose one could argue that, the CPython implementation is the language specification. (And I seem to recall hearing that notion somewhere years ago.) In which case, it would not be possible to freeze development of the language without freezing the implementation as well. There are various reasons I wholeheartedly disagree with such a characterization, but I guess there's some self-consistency there at least.

> ceasing development of the language Python 2 is wholly sensible, ceasing development of the implementation Python 2 (CPython specifically) is not

Development of CPython 2 has ended, bugs and all. It's past its end of life, this is well known and has been known for a long time. Any remaining bugs are the problem of the users, not the responsibility of the former developers.

Sure people will fork it and do stuff with those forks, but those will no longer be new versions of CPython, they will be new versions of some-fork-of-CPython.

This is clear, but does the cpython team want to maintain Python2? If not, either it’s time to use an alternative or move on....

> So your claim is that "Python 2" is a language spec, not an implementation? And that there will be future releases of this language spec in the future?

PyPy maintains a Python 2 implementation and will continue to do so.

> "Python 2" is a language spec

Cython, IronPython, Brython, Stackless Python, Nuitka, etc.

Does anyone know what the main reason is for not updating from python 2? I'm genuinely curious as I don't really know any modules that won't work under Python 3 and I can't really come up with any other blocking changes that would make upgrading that hard.

I did the work for a reasonably sized project recently - a few hundred thousand LOC. It was long, boring, risky work. Let me rattle off some of the tasks.

Audit all strings coming in and going out for encoding issues. Update all dependencies to their python 3 equivalent. Replace dependencies that hadn’t been updated (typically older django dependencies). Use python-future to bulk update incompatibilities. Changes to metaclasses were annoying. Force all uses of pickle to use protocol version 2. I documented some more during the migration on Twitter https://twitter.com/jarshwah/status/1209381850822496256?s=21

We began getting the code base into a compatible position about 1.5 years earlier. A final push of 3-4 weeks of work got it over the line, with many bug fixes after the deployment.

Other older larger systems will have similar problems at a larger scale.

This isn’t a condemnation by the way. Python 3 is better. The only reason we held out so long was because of the business justification. Once we couldn’t wait any longer it got prioritised.

That's all uncannily similar to my experience porting our ~350k LOC app.

"It's a dirty job, but someone's got to do it".

Many internal tools for one, platforms, etc. Hard to tell.

One industry example is https://vfxplatform.com/ - they just (this year) moved to Python3, but with some delays, from the site:

    The move to Python 3 was delayed from CY2019 to CY2020 due to:

    No supported combination of Qt 5.6, Python 3 and PySide 2 so Qt first needed to be upgraded.
    Upgrade of both Qt and Python in the same year was too large a commitment for software vendors and large studios.
    Python 3 in CY2020 is a firm commitment, it will be a required upgrade as Python 2 will no longer be supported beyond 2020. Software vendors are strongly encouraged to provide a tech preview release in 2019 to help studios with testing during their Python migration efforts.

The active development of Python 2.7 stopped in 2015, that was the time to start migrating. Seems like this application would never updated if 2 wasn't EOL.

There are still Classic Visual BASIC programs out there that haven't been ported to VB.Net or C# yet because of how huge they are and how hard it is to code that they cannot afford to hire developers to do it for them. The same is true of many old languages like COBOL.

I heard that some places still using Turbo Pascal for DOS and have to stick with 32 bit machines because 64 bit can't run 16 bit DOS code.

Yes, and you similarly can continue using python 2.7.18 for next 10 years, no one expects Microsoft to continue releasing new versions of the classic VB. A lot of python users have weird expectations.

Lot’s of code is simply unmaintained. The guy that wrote it is gone, it’s still running fine so nobody is touching it. Businesses don’t want to take the risk and spend the money to upgrade it. Maybe you don’t realize the insane amount of code that is in this state!!

If you depend on an unmaintained codebase where the original developers are no longer available, then that's a substantial business risk by itself.

Too many software development projects are treated as one-off events where people commission them and assume they will work forever without updates. Software requires maintenance, and people who commission software development projects without planning on how they are going to be maintained in the future are taking on risk. Any risk involved in updating that abandoned code in future is a consequence of that decision.

In some ways that's the whole point of code. You want to get to forget about it doing all these things without you.

If that's the case then 2.7.18 will continue working and probably it is a bad idea to port it. A lot of work for minimum gain.

But if you actively changing the code, the maintenance will get more and more expensive. With packages dropping python 2 support if you discover a bug in one of your dependencies and fix is in package that no longer work on python 2 you'll need to backport the fix (and maintain your fork) or migrate your code.

This is in part due to a lack of foresight, but you can run into all sorts of weird issues that you'd never think of, like this one: we have a feature in our REST API that can return lists of items as CSV instead of JSON (yes, I know, it sounds weird). It requires no effort from our backend services; the api proxy takes care of it. Unfortunately, something changed with dict enumeration order between python 2 and 3, and so when we first tried to upgrade, the CSV files being spit out had a new column ordering, which of course would have broken customer code that relied on it.

The string-handling changes, while necessary, are also a bear to deal with. Since python is dynamically typed, you need to work to find all the places where you need to add a ".decode()" or ".encode()". If you don't have excellent test coverage already, you're going to miss some, and it'll be a game of whack-a-mole until you get them all... assuming you have actually gotten them all.

> something changed with dict enumeration order between python 2 and 3

Dicts were by definition unordered until Python 3.7 [0], so you were relying on undefined behaviour. If you need an ordered dictionary and support Python 3.6 or below, you should use OrderedDict [1].

[0] https://mail.python.org/pipermail/python-dev/2017-December/1...

[1] https://docs.python.org/3/library/collections.html#collectio...

> something changed with dict enumeration order between python 2 and 3

Enumeration order in dict keys was never guaranteed (until 2019), even on 2. So basically that code relied on undocumented cpython behaviour that was strongly advised against, i.e. it was broken already. 3 simply made the brokenness more visible.

I imagine the reason for not upgrading from Python 2 is the same reason you don't upgrade your car just because there's a new model out. (Or maybe you're the type who does, but I guess you can hopefully understand why others don't do that.)

The Oil shell, at least, reverted to (a forked version of) 2 after upgrading to 3. http://www.oilshell.org/blog/2018/03/04.html#faq

How has that project not been sued into oblivion by Royal Dutch Shell yet...?

I think the biggest issue seems to be that you need to migrate the whole thing at once. If it could be done incrementally it would be easier.

Although there are ways, you can still incrementally adapt code base to work on both pythons. Also pylint with py3k option, mypy can help. There's also six packages, but many people seem to had good luck with futurize.

There's also something that I tried a while ago and it surprisingly worked (although it might not work that well on larger codebase?), basically you can use Cython (not to be confused with CPython) to compile Python 2 code and then include it in python 3, this would enable migration file by file.

Personally, the main software I use at $DAY_JOB doesn't support python 3 yet.

I'm not looking forward to the scramble when it does upgrade, as we're using a ton of community modules that may or may not be abandoned.

Your existing code work perfectly for now. Management failed to understand while they need to budget a team to upgrade when the project doesn't bring anything new to the table.

Our migration to Python 3 occurred with a new generation of the product. No new development is happening on the Python 2 product, and customers are being migrated off. The vendor who runs our old product (GAE) has promised ongoing Python 2 support, so there’s literally no reason to spend the time or money to migrate it, no matter how long it takes for the last customer to get off the old product.

Because giant codebase that would take effort++++ to do so for no perceived gain, with possible addition of new bugs.

> effort++++

effort += 1 + 1

Ported to Python3 for you.

Clearly c got it wrong. The number of + should have been increment so e->e + 1 should be e+ e+->e+ +1 should be e++ ...

Correction:

  effort = (effort := effort + 1) + 1

Executable pseudocode!

That's how we get decades old cobol codebases that nobody understands.

In my work, it's software that offers a Python 2 module for scripting. I tried the naïve "upgrade" of copying the module into my Python 3 module library, but no dice. The software checked the version of Python, saw it wasn't 2.7.10 (yeah), and raised an error.

Some of the older hosting services still support only Python 2, or some early crappy version of Python 3.

The same reasons that an even larger codebase never upgraded from Visual Basic 6.

Because there's nothing to update to. Everyone who was working Python circa 2.7.0 has switched to working on a new, different language which they insist on misleadingly calling "Python 3" rather than come up with a new name like the Perl -> Raku folks did.

I think you are trolling, but in case you aren't; can you elaborate what is so different in Python 3? Granted, it is not a drop-in replacement but it is 99.9% the same thing.

There's a lot of code out there that's important, but not worth porting to a new language (which Python 3 is).

I think describing it as porting to a new language is misleading: on most of my projects, most of the work is a few minutes — run modernize/futurize, check the tests, etc. If the original developers were really sloppy about how they handle encoding, it can take longer but most of the problems I've seen have very little to do with Python rather than the fact that something still running Python 2 likely has significant technical debt issues — especially things like not having test coverage which make it a lot harder to ship changes.

> likely has significant technical debt issues

You're not wrong, but as a practical matter, that pretty much describes our entire industry.

Completely agreed — I would just argue that the “Python 2 vs. 3” argument is a distraction. Java hasn't had as breaking a change but there are still a ton of places running Java 6 or 7 because they like skimping on developers more than getting security updates.

Our product took about a year to port, from getting the go-ahead to the eventual production switch. Running modernize / futurize was like 0.01% of the work.

You're right though, we're fighting our way out of technical debt and switching to python 3 was absolutely necessary. It's forced us to sort out a lot of sketchy string / bytes handling. We do, mercifully, have ~90% test coverage.

A year is astronomical. Does it involve writing tests that did not exist before?

I don't remember us writing any new tests specifically for the python 3 port.

One issue with the test suite was that it made heavy use of a thing called django_any that isn't supported in python 3, so decided to replace it with Factory Boy. We have about 500 django models that needed new factories. Factory Boy works quite differently and it was a lot of work to make the factories behave similarly to the old ones where possible, and update most of our ~4000 tests for the new behaviour.

So that was one issue. It was tempting to just patch django_any, but we decided to tackle the technical debt instead.

> something still running Python 2 likely has significant technical debt issues

Most codebases basically.

There isn’t anything inherently wrong with still using 2.7.x . Just don’t expect updates. For new code using 3.7 is probably the best bet at this time.

I thought there won't be any more security updates?

Red Hat, Ubuntu, etc. are going to support Python 2 for the duration of the operating system releases which shipped it. I would assume that Anaconda, et al. will have similar options for paid customers.

Red Hat has committed to keeping Python 2 on life support until 2024 as part of Red Hat Enterprise Linux 8 [1] so you can get security fixes for Python 2 until then if you use CentOS 8.

Canonical will not provide long term support for Python 2 as part of Ubuntu 20.04 LTS. In Ubuntu 20.04, Python 2 is a "universe" package [2] that does not receive updates by Canonical. This means that the you will only get Python 2 security update guarantees with Ubuntu is on 18.04 LTS until April 2023.

Debian is making an active effort [3] to remove Python 2 and packages that depend on it for its next release. It'll likely support Python 2 as part of Debian Buster until 2024.

Note that if you're reading this to delay your move to Python 3 by another few years, you're doing it wrong. This list shows even all slow enterprise-y distros have a deadline for Python 2, not that you can stretch your stuff for a couple of more years :)

[1]: https://access.redhat.com/solutions/4455511

[2]: https://packages.ubuntu.com/focal/python2

[3]: https://wiki.debian.org/Python/2Removal

I believe the biggest thing to worry is your application dependencies, if you also depend on packages that come with your system then probably fine (although I noticed that these are largely ignored even if there bugfixes they don't update them)

Otherwise even if your python has security patches for next 4 years, it won't do you any good when you find a bug in one of your dependencies and bugfix is in a version that's python 3 only

Thank you for providing the extra details — I especially agree with your conclusion: go to your boss and say “even if we pay, we're looking at a drop dead date no later than 2024”.

So? Im sure my non internet facing scripts care

Distro maintainers will be patching security bugs for the foreseeable feature. Do you seriously think if there is a security bug found today Debian maintainers will be like "ah, though luck, I suppose people need to upgrade to py3"?..

Broadly, distros have been ripping out Python2 left and right in advance of 2020. Debian may have a longer support cycle than most and still have Python2 in stable or oldstable.

> Do you seriously think if there is a security bug found today Debian maintainers will be like "ah, though luck, I suppose people need to upgrade to py3"?..

> During DebConf19 we¹ have tried to figure out how to manage Python 2 and PyPy module removal from Debian and below is our proposal. [0]

Debian are in the midst of a large project [1] to remove Python 2 as quickly as they possibly can. Whilst some bugfixes may happen, Debian are already telling you in no uncertain terms:

> port upstream package to python3

> remove any Python 2 use

[0] https://lists.debian.org/debian-python/2019/07/msg00080.html

[1] https://wiki.debian.org/Python/2Removal

Python's open source. Anyone can do security updates. Teams at RedHat, Debian, Oracle, etc, will be doing security updates for many decades I'm sure. You may have to pay.

Can't do it under the "Python" trademark name, though.

A huge amount of the work of distro maintainers is actually just this kind of backporting and applying security fixes. You're right that technically, the python foundation (or whoever owns the trademark) could come after redhat for making these kinds of changes but it's very doubtful they would.

If redhat decided to add new features to python 2.7, I'm sure the PSF would make a stink

You can, that's what RedHat is doing. It will be still Python 2.7.18 + security patches.

You probably confusing it with Tauton (a Python 2.7 with backported Python 3 features) that tried to place itself as Python 2.8. By backporting these changes they created essentially 3rd version of Python that was incompatible with other 2.

Oh no, it’s the whole “IceWeasel” fiasco all over again.

Maybe they could call the new 2.7 interpreter IceSnake.

Not sure that's true?

> As such, stating accurately that software ... is compatible with the Python programming language, or that it contains the Python programming language, is always allowed.

There weren't many before this, either.

3.8 seems to have some new keywords and jazz introduced, so might be better to start with that instead of 3.7 for new projects.

3.8 seems to be much more twitchy about exact versions of dependencies, so I've had problems running the AWS cli stuff on 3.8 at times, because there's no set of non-conflicting dependencies. (oftentimes due to minor/patch level version mismatches)

I keep having issues with 3.8 and many dependencies. Two months back, I started out a new project in 3.8 and two days in was downgrading it due to compatibility problems with Pillow and a couple others.

What typically is happening is that a wheel package was missing. When that happens python tries to compile the package, to do that it requires extra dependencies, such as compiler, python-devel, and other *-devel packages, because they weren't available it failed. This is very common when a new major version is released, it requires authors of the C based packages to build wheels to make installation easier and not requiring extra dependencies.

Looks like Pillow has wheel for 3.8 now since April 2nd, so might work now (no compilation is needed). I don't know other packages so can't check them. Psycopg2 would probably be another one with this issue (also fixed on April 6th).

[1] https://pypi.org/project/Pillow/#files look for cp38 wheel files.

The CentOS base yum repository only has python 3.6.x. I hope they will make later 3.x branches available for installation in the near future.

CentOS / RHEL 8.2 will ship with 3.8.

Just discovered a week or two ago a pretty important part of one of our internal systems is running on python 2.6

Caveat about 2.7: HTTPS switches from default-non-validating to default-validating. If your internal system uses SSL, and you have a bunch of self-signed certs, for example.

I think we're prooobably good there, this system is primarily slinging around and transforming data between databases.

The far bigger worry would be unexpected encoding issues corrupting the data when going from 2->3.

> Even know one researcher using it for a new project.

Knowing researchers (at least in academia), they're the last people I would expect to change.

And those are least worrying, once they publish their thesis, they don't give a damn about their code anymore and it just dies natural death ;)

"So?" said COBOL.

"Hold my beer!" - COBOL

We've switched off Python 2, but I really miss it. For a glorious several years we were done with "production has Python X.Y, but the code needs X.Z" and always chasing the latest minor version on LTS OS releases.

But now we're back on that treadmill...

How long does python upgrading take you? I think Python 3.7 to 3.8 was change my docker version and push to CI... 5 minutes later it told me all tests passed and I deployed to production??

I upgraded from 3.7 to 3.8 and found out they removed a function in the time package required by crypto.

I do strongly feel that removing functions from the core library is a huge no-no for point releases.

From afar, Python 3.x releases seem more like major updates than point releases. There’s entirely blocks of functionality/syntax between 3.x releases.

Except it is mostly additive.

Adding new features will seldom break old stuff. It is the removing part that is hard.

(With the exception being when like a variable broke because it became a keyword, but if you made a variable something like async I am not sure you are entirely innocent).

You're supposed to pay attention to DeprecationWarnings.

The problem is that the entire ecosystem needs to pay attention to such warnings and it doesn't happen. As a result, these changes end up breaking code in places the program authors never touched.

That's true in one respect, but you as an end user can use them to know not to upgrade before your deps have changed what they need to.

What? You upgrade your dependencies when there's a feature or improvement you need. Not because they don't have some warnings. Who cares?

The old version you implicitly claim is better isn't, it just doesn't have the warnings from their dependencies in place yet.

In my experience DeprecationWarnings get turned off so frequently because I'm not forking big library to fix all of it's Deprecated uses of their dependencies.

The warnings are useless if they're not from my code, so they get turned off once globally.

They are still important. Because even if dependency has this warning, it will still break your code.

"The function time.clock() has been removed, after having been deprecated since Python 3.3"

Python 3.3 was release in 2012. You've had 8 years.

Well, technically we already moved to 3.8, after which needed a library that only works up until 3.7.

It would be nice if library developers kept their code up to date, but that doesn't always happen. Python core devs know this; well all know this, yet they consciously screw with the core libraries with the principle, caveat emptor.

I don't understand why they don't ear-mark these changes for 4.0. These kind of things are a universal frustration with the community and they are so easily avoidable.

What difference would it make whether the change had been called 3.8 or 4.0?

I'm confused, who exactly has had 8 years to do what? Should he have dumped the crypto module because it was using a deprecated feature?

The reason the crypto module was using this particular deprecated feature is that it hasn't been updated at all in 8 years.

The OP should have dropped it because it's unmaintained, and a maintained replacement has existed for a long time: https://cryptography.io/

This is an especially important consideration for security-critical libraries like cryptographic libraries.

See, when you explain what's wrong it's so much better than just blaming the victim with "You've had 8 years"!

this is text from maual:

> Deprecated since version 3.3, will be removed in version 3.8: The behaviour of this function depends on the platform: use perf_counter() or process_time() instead, depending on your requirements, to have a well defined behaviour.

I would be wary of any crypto library that continued to work with a warning for 8 years and no one bothered to fix it. Most likely no one was maintaining it.

Perl does it as well, but then you've only got point releases because it's stuck at 5.

I think at this point they should jump to perl7 and resume normal versioning

Python 3.7 to 3.8 broke some stuff. I was pretty annoyed to discover (the hard way) that e.g. this wouldn't run anymore:

  python3 -c "import ast; print(ast.arguments([]).args)"

It seemed completely unnecessary too. They could've just kept the structure the same across minor versions...

I think that probably lesson learned from python 2 -> 3 migration. It's probably better to introduce many small breaks than a big one?

The python despite having 3 digit versions apparently is not following semantic versioning. The major version number generally had not much significance (the python 3 was exception, 1->2 and what they are assuring us 3->4 won't be a big change)

"Better" depends on your agenda. If it's to shove the next major version down users' throats, maybe it's easier to swallow small bites (bytes?) than one giant cow. If it's to provide a predictable and pleasant user experience, I'm less confident this is the right approach.

So I did look it up, they do provide a warning though (in the first paragraph):

"The ast module helps Python applications to process trees of the Python abstract syntax grammar. The abstract syntax itself might change with each Python release; this module helps to find out programmatically what the current grammar looks like."

This module is a bit special, because as they say, it supposed to reflect current python grammar. If they change grammar and didn't make it reflect it it would lead to different kinds of issues.

It's one thing to extend the AST and add a couple children because you added new syntax or something. That's what I think most sane people would expect from reading that quote as a language evolves. So yeah, don't assume a node has 4 children when it might have 5+ tomorrow. But it's another thing to outright remove a child entirely and to be so vague about it too. (They didn't even have it be a sentinel value like None, or even care to leave a useful error message in the stack trace! The message makes you think the code was broken all along and simply never triggered before.)

Lest you think informing users more clearly is a foreign concept to them, look at the dis module [1] for comparison. They're extremely clear the whole thing is implementation detail of CPython. If anything, after reading that, one would think your conclusion would be "ah, I should program against the AST then, not the bytecode". So you do that and then you're greeted with this nonsense! Obviously it's your fault for assuming there's anything stable to program against across minor releases.

[1] https://docs.python.org/3/library/dis.html

That is what is great about having a good test setup. You would know pretty quick if stuff broke. Which has been basically nothing for me (3.6 -> 3.7 -> 3.8)

To be clear, the pain here was hunting down the root cause and fixing it when it shouldn't have broken in the first place, not the speed of discovering the problem after the code had already broken.

You'd be amazed out how many stale docker vm's are out there running old python versions - docker doesn't solve this issue and in fact I've seen docker images get updated less rather than more often ironically!

> docker doesn't solve this issue

Well. The container doesn't magically update itself. But FROM python:3.6 -> FROM python:3.8 doesn't seem that hard. Then again a flexible package manager (with recent repos) can do this as well.

For a while my company was considering docker precisely to keep an old python app (described as python 1.5) running. The dependencies of that app haven't been maintained in a decade, will never be upgraded and will likely disappear soon as the hosts remove their mercurial repos. Having the base machine running something modern with the old python stuff in a docker image seemed like a promising solution but the stuck with RHEL7 instead.

Thankfully it's no longer my problem.

I am using my own docker image though.

I control what is installed on it.

But what version does production have, and all the other servers and projects?

That wouldn't matter if you're using docker, unless I'm missing something here.

Exactly. Each of my apps has their own python runtime. Running on top of kubernetes.

Man in 2020 if you are SSHing between boxes and upgrading python by hand, you really should invest in some devops. Even 5 hours a week.....

We have VMs and deploy using Ansible, devops doesn't have to mean running everything in containers using Kubernetes.

Our applications are mostly monoliths and the number of servers they use is quite constant.

But it does mean our Python version really only changes when we switch to the next Ubuntu LTR, and isn't always the same between different projects.

I hope you realize not everyone running python is running web apps. There are huge companies that use python for automation and they all run from some computer grid storage (usually NFS) and are imported in the same environment as the client instead of having a server side environment separated from the client side environment.

98% of my containers are not web apps.

Still a lot easier to manage as docker volumes in k8s.

Yes, I was only thinking of my own situation (we use Docker in development, but not in production except for one project).

It's not how long it takes me, it's how long it takes Ubuntu. Usually, a few years...

If I don't want to use OS packages, I have to decide to package, build, deploy, and maintain the Python packages, often in a way that separates it from the system Python package, while also managing any Python libraries we need (some of which may be C extensions and require compiling).

Whenever possible, I like to leverage upstream packaging so that I don't have to track security updates on my own.

I say this as someone who used to maintain the official Python.org RPM packages.

This is underestimating the number of packages not ported yet. Last time I checked, pygame and opencv were among those.

> Last time I checked

pygame supports python3 since 2016, python-opencv since 2018 at least

The comment I'm replying to is about the transition from 3.7 to 3.8

> I think Python 3.7 to 3.8 was change my docker version and push to CI

You could just stay on 2.7 and let the others have their "Python"

I still occasionally mistype `print s`.

RIP, python2, you will be missed. And a big thank you to all the contributers who have kept it alive for so long!

I still feel very wrong whenever I have to use print(), it was so much better without ()

Agreed. It was so less painful to add and remove for debugging. Which is it's only real use-case.

Yes - it is incredible how big a deal this is. I never would have thought a few characters would mean so much to me. Ah well... :)

Is this the most infamous failure of a version upgrade ever in software history?

How is it a failure? When they were talking about Python 3 at PyCon 2008 and 2009, it was expected that the migration would take this long. Folks knew migrating would be long and hard, and that the approach would need to be revised.

It's only relatively recently that Python 3 passed a certain threshold that made deprecation of Python 2 something that could actually be executed upon. Had the growth been slower in the last five years I think the timeline would have been different.

If I was going to critique Python "failures" I think the bigger target for me would be the integration of async concepts into the platform and language. This could have been handled more smoothly and with better community cohesion, but most HNers only have a superficial understanding, i.e the GIL is bad and Python can't compete with Go.

The migration succeeded but the original plan didn't. People were supposed to write Python 2 and generate Python 3.[1] What worked was making it possible to have a single code base.

[1] http://python-notes.curiousefficiency.org/en/latest/python3/...

Agreed, all successful migrations I've seen started early and consistently used `from __future__` and python-future to write compatible code over a few years. This incremental approach almost made it trivial.

How is it a failure?

It was an immense amount of work for a fairly small payoff.

Wasn't the original timeline meant to be completed in 2014?

The original timeline was very much a guess. By the time that people got into the details in 2010-2014, it was clearer it would take much longer. This doesn't make it failure unless you lost a bet with someone on the timeline.

Upgrading a programming language with breaking changes is hard.

Ask the PERL and PHP communities how hard it is. PHP 6 ended up getting dumpstered and the “next” version of PERL was so different they had to change the name and the community never really recovered. Python has at least grown/ survived through the upgrade.

Perl 6 may be a contender, now so different that it's renamed to Raku.

Python was roughly # 20 when “P3K” came out.

Python is now roughly #1.

Let the data tell the story, and not the other way around.

Not even close.

Also, it takes pretty vivid imagination / stout obstinance to characterize vast majority of pre-existing libs/projects and almost all new project starts using new version as a "failure".

Bunch of whining and FUD does not equate to failure.

> Not even close.

What example would you point to as a bigger failure?

All of the C versions post 99 or so probably have worse adoption. New versions of ALGOL, Visual Basic, Cobol. Being infamous isn't failure being unknown is a failure. Raku is a success I never have used Perl but I checked out Raku seems alright.

Perl 6, which failed so hard it's not called Perl anymore

yeah, but to be fair people had a gut-hate reaction to the word 'perl' before the update.

Perl ranks up there at the top of 'lists of programming languages people love to hate.'

ALGOL 68?

Python 3 is an enormous success so I don't know what you could be referring to.

Everyone who ships software knows that there is always a very small but disproportionately vocal set of malcontents who can't accept changes; that's their issue, not anyone else's.

It was mostly a success. People could continue to use the old version for over a decade after release of the new one if they wanted to, and the end result is that Python is now one of the most popular programming languages.

They've also been given more than enough guidance to prepare for a smooth upgrade.

Perl 6?

People actually use Python 3, though. The Python developers succeeding in forking the language and community. No one really uses Perl 6, as far as I know. You're right that Perl6 is another great example of a bad language fork decision.

The community is certainly not that much into Raku, but from this past thread https://news.ycombinator.com/item?id=22906703

> On the #raku channel on Freenode, several users are using Raku in production.

Some vocal members of the Perl community have not been into Perl 6. That didn't change with the name change. But if anything, the change in name should make clear that even though Raku is clearly a programming language in the Perl lineage, it is not aiming specifically for the Perl community. Anymore, at least.

I feel like the Python 3 experience could have been predicted by looking at the Perl 6 experience.

It's not too late to change the name of Python 3, just as Perl 6 has changed its name to Raku (https://raku.org using the #rakulang tag on social media). :-)

If only we all could "fail" so spectacularly. :D

php has entered the chat

since php6 was never released, it cannot be a failure.

Python 2 had a great run but python 3 is more than ready for prime time. It was pretty bad for a while but since 3.6 it’s been awesome!

Agree, actually it started being usable since 3.3 (and each following version was better than last one), but 3.6 was when there was no longer anything that python 2 was better at.

Finally. Please let it die.

Languages that receive wide adoption never die.

Anyone who thinks they do should go sign up for some COBOL training so they can grab those sweet New Jersey contracts. ;)

I'm sorry, but characterizing Python 2 as a completely different language than Python 3 just makes me laugh. Is American English a completely different language than British English? (I realize this isn't the best comparison, just trying to make a point.)

Like I said, I acknowledge there were costs to transitioning. Python 3 was pretty rough around the edges upon first release. But that first release was in 2008 ...

I realize not everyone needs to upgrade, and that's totally fine. But how people internally justify the cost of keeping Python 2 going when the momentum and most of the community is with Python 3 I cannot understand. Unless you're in a very inflexible corporate setting ...

>Is American English a completely different language than British English? (I realize this isn't the best comparison, just trying to make a point.)

Which one of those two languages will give me No Module Found when I go to import openCV?

Different features, different uses, mutually exclusive, python2 and 3 are different languages.

I think your reply is overly pedantic, but I guess I asked for it, didn't I.

Both? You'd import cv2, which works in either.

It is a completely different language - it is syntax incompatible. Human languages are a very poor comparison to computer languages.

It's just a tough sell to manually update a huge pile of code that works just fine as-is. No new features will be introduced. No bugs fixed, while some new ones may pop up. There's a lot of manual work in porting, the automated tools only cover the basics. Your downstream customers receive very little benefit for this effort, which makes it very hard to justify spending the engineering resources on it.

> No bugs fixed

At this point, I've helped port multiple code-bases in multiple companies. Porting to Python 3 actually can and does fix some bugs, as it generally forces the code to start handling text semi-correctly, instead of just hoping the bytes go through and it all works out.

> Your downstream customers receive very little benefit for this effort,

They do, actually. Again, while I was porting those same code-bases, I'm working with a team of engineers who (and along with myself) are also still adding code to meet other incoming requirements. And in my experience, there was a fair bit of "Boy, it'd be nice if X were easier in Python!" where X is something that is easy in Python … 3. The question would have often gone unanswered had we not had devs experienced with Python 3 on the team.

Porting to 3 gets you all the additions that have come to Python that haven't been backported. A better standard library, syntax that better supports you, etc., translate to better productivity for devs as they are now equipped with better tools. This is only going to get worse as libraries drop support for 2.

(And while many things have been backported into third-party libraries, not all of them have, and in particular, syntax changes. And the existing backports are actually quite useful during the process of porting: I can change the code to conditionally depend on the backport library in 2, and use the real deal in 3.)

I appreciate what you folks are saying, but from my perspective this is exactly what I mean by the absence of a "good technical reason".

I also appreciate that there are a small number of projects (e.g. PyPy) that probably do require some of the special sauce around CPython 2.

But for everyone else ... come on folks. We're developers and engineers, we're supposed to be the creative builders of tomorrow and all that stuff. Is this really so hard?

Modern Cobol isn't the COBOL of the 1950s.

I just don't understand this sentiment. If some stranger wants to continue using Python 2, why does that bother you?

Why did I not see this sentiment for C89 vs C99 for example?

If a stranger wants to run anything on their system, alone, never touching anything else and never coming into contact, whatever. But I very much look forward to a world where "Python" == "Python 3" precisely because it keeps impacting me. If I use a package that continues to use python2, I have to keep a python2 installation on my system (complete with some set of modules). Meanwhile, I can use the very latest, 100% maintained GCC or Clang to compile any C standard version, so I don't think that's a good comparison.

> Meanwhile, I can use the very latest, 100% maintained GCC or Clang to compile any C standard version, so I don't think that's a good comparison.

So it's PSF's fault for not providing a -std=py2 option? I guess that's a sensible way to look at it.

I would hesitate to call them at fault per se, but yes I think if such an option has been provided, particularly at a fine-grained level so that modules could be moved over piecemeal, we would not have had nearly as much difficulty with this transition. On the one hand, removing some of the pressure would have meant that the old versions stayed around longer, but if the system genuinely works and doesn't hold up the evolution of the overall project, I'm not sure that I see a problem with that.

Mostly just because of all the oxygen this debate seems to suck up, and the lack of reasonableness around it. It also bothers me because some of this nonsense seemed to contribute to Guido stepping down as BDFL, which I think was a Bad Thing.

Let me be clear: If people want to use Python 2 they can go nuts. They can even pretend it's a completely new language if they want to. If there's enough of them I'm sure it can be done. Just stop asking the community to support it, or the rest of us to care.

Imagine if it was the day of the Super Bowl, and you wanted to go out to a sports bar that was going to be airing the Super Bowl. So you go out and find a bar that says they're throwing a Super Bowl party, and go in and settle in, and when it comes time for kickoff you realize that the bar is actually airing the Super Bowl from 10 years ago. Like, it would have been fine if they had called the party a 'Super Bowl from 10 years ago' party, and for some reason there's this part of the bar that seems to be all on board with rewatching the same Super Bowl from 10 years ago every year and frankly seems confused why the rest of the bar is confused and angry about it (I mean it's still the Super Bowl right?).

Thanks for your perspective. Here's an analogy which illustrates a slightly different perspective: Imagine that 10 years ago, the NFL introduced a 50-yard indoor variant of the game, but saw no need to pick a different name for it, because obviously everyone would abandon the existing 100-yard stadium variant.

Guess what's going to happen for the next 10+ years?

If some stranger wants to open a hole in the ozone layer, why does that bother you?

My personal thoughts on the matter aside, here is an informative post from Brett Cannon on why the Python 2/3 split happened:

https://snarky.ca/why-python-3-exists/

That's what happened to fork the language, but really glosses on the "why." Paraphrased: "People were too lazy about using the unicode type and bytes type, so we forced all string constants to be unicode." I think there is some cognitive dissonance between thinking this fixes lazy programmers and expecting codebases to be rewritten in the new language. (I.e., by who? Those same programmers too lazy to use the unicode type?)

Also, all the world wasn't unicode at the time and still isn't (unfortunately, IMO, but still the case). We still have customers using various non-Unicode encodings and not all characters have a 1:1 encoding/decoding round-trip through Unicode.

The article does say people get lazy or performance hungry but the theme of that section is people making mistakes.

The part about Unicode is about how much Python 2 code accidentally only handled ASCII. Python 2 requires handling other encodings as bytes anyway.

> That's what happened to fork the language, but really glosses on the "why." Paraphrased: "People were too lazy about using the unicode type and bytes type, so we forced all string constants to be unicode."

I strongly disagree. I actually tried using unicode correctly in python 2, it is actually quite hard. It's similar to using memory safely in C, it is possible, but there are many gotchas.

In python 3 all of this is done with no effort.

I'm curious if there's going to be some community supported fork.

Software using Python 2.7 won't magically disappear and I believe there is going to be a demand after fixes.

(I know about Tauthon but I don't think it applies here ...)

I think it's really unlikely because the primary reason people are still on Python 2 is that they don't have development time to upgrade. If they had enough time to support a community fork, they could just upgrade to Python 3 and spend the saved developer hours on something more important to their projects.

Collectively support maintenance of Python is much less work (and less risk) than to port all existing production Python 2 applications though.

The problem is distribution: this might be true for the entire world but the time demands to start and maintain such an effort are larger than the cost of updating all but the largest projects.

That would be great imo. Then the rest of us who are happy with Python 3 can just get on with life.

I'm willing to bet that the demand is not really there and most of this is just kvetching. Should have happened a long time ago.

I'm not aware of any good technical reason to still be using Python 2. There is the "I don't have time to upgrade" argument, and I recognize the cost is real. But now we'll see at last if people are willing to stick up for Python 2 more than just figuratively.

Dependencies in computer graphics is still a big holdout unfortunately.

The industry is still not in the process of porting major pieces of the ecosystem.

It’ll be another 3-5 years i estimate

There is Tauthon[0] that (they said) also backported several features in non-backwards compatibility breaking ways, but the last commit to the repo was from December. Perhaps after running it for 3-4 years (and faced hostility from some Python devs - i remember at some point being downvoted just for mentioning it on Reddit), the maintainer got tired of it?

[0] https://github.com/naftaliharris/tauthon

might also be indicative of how many people yelling for a fork were actually willing to put in the time to maintain it or pay for it?

(most places i've worked at, when we do the calculations for the business it's much cheaper/less risky just to upgrade to python 3, rather than to support python 2 or switch languages)

I guess PyPy is another 2.x language implementation outside of CPython that might continue to support 2.x. However, there's a reason it isn't universally used to run CPython code, so that may not be appealing as an upgrade path from CPython 2.7.

https://www.activestate.com/products/python/python-2-7/

This is the Python dev community telling people that it isn't supported.

Demand for fixes will occur but it would probably best be handled by a third party that would specialize in that or developing it internally.

RH, Canonical, SuSE etc probably will be trading patches, but I doubt they will organize to do independent "releases" anymore.

I love so much about Python 3.

Python 2 has paid the bills for over a decade, but I love writing Python 3.8 so much more. I even love the walrus operator. We’re in the process of upgrading, finally.

I thought the last release was already past; wasn't the python 2 series supposed to have ended on January 1st? Or did I misunderstand that?

2.7.18 was the planned final production release [1]:

> As of January 1st, 2020 no new bug reports, fixes, or changes will be made to Python 2, and Python 2 is no longer supported. We have not yet released the few changes made between when we released Python 2.7.17 (on October 19th, 2019) and January 1st. As a service to the community, we will bundle those fixes (and only those fixes) and release a 2.7.18. We plan on doing that in April 2020

[1] Sunsetting Python 2: https://www.python.org/doc/sunset-python-2/

[2] When is the last release of Python 2.7 coming out?: http://python-notes.curiousefficiency.org/en/latest/python3/...

They stopped making changes on January 1st 2020

https://github.com/python/cpython/commits/2.7

This is the release of the changes up to that date, since they don't release instantly.

Final fixes from 2019 that got stuck...

https://adtmag.com/articles/2020/01/09/python-2-end-of-life....

Python 2 ended support on January 1'st however critical bugs reported before then continued to be worked on until this last release. All Python foundation development has now concluded and any further bug fixes will have to either come from the community or another entity.

Apparently it takes 4 months of planning to create a release. 2.7.18 was not accepting any changes after Dec 31, 2019.

Trying to click the release notes for it on https://www.python.org/downloads/ seems to do nothing. I'm working on legacy apps running on 2.7 so I'm eager to see what's included.

As far as I can see this should be the list of changes: https://github.com/python/cpython/blob/v2.7.18/Misc/NEWS.d/2...

Has anyone seen a recent survey of which version of python developers are starting projects with?

I start every project on the latest widely compatible release. atm, that's 3.7 or 3.8.

Once you're using containers, shared build tooling, and enforcing linting/testing, the version you use doesn't matter as much, imo. Unless you're using an advanced feature that's still developing (rare..only seen it a few times), most projects can upgrade minor version with zero codebase changes. Just lock a new environment, format code, and push to CI.

It's probably worth considering the number of former Python developers who were bit by this or other breaking changes in 3.x and have moved on from the language entirely.

I haven't heard of anyone starting a new project in 2 in several years.

See: https://www.jetbrains.com/lp/devecosystem-2019/python/

Mac vim(1) still requires Python-2. I'm not holding my breath.

I am still on Python 2 due to complex vendor reasons. Probably will be for a couple more years. I wish folks were a little more understanding that edge business cases exist that they might not have considered; it isn't all greenfield development.

Free support for ten years! Doesn't get any better than that.

Wish I could get pip running. That'd be nice.

> Python 2.7.18, the "last" release of Python 2

Fix't it for ya.

I expect many forks.

But none that are actually going to be maintained in any meaningful way. Maintaining Python 2 is a huge job, which is also why the Python core team can't do both 2 and 3 at the same time.

Well, the team that maintains the fork wont need to do both 2 and 3 at the same time either, just 2 :-P.

No but they will have to sell the "it's not official Python 2" line.

I didn't realize how much effort it would be. Indeed, I think that comports with what I've seen from other forks - it takes a huge effort to keep it going.

You mean pitchforks?

And now for python 2.8. From somewhere, if not python.org.

Caring for dwindling groups of increasingly eccentric users sucks. That doesn't mean nobody will do it, especially at first, and especially if there's money in it, but don't expect such maintenance efforts to have much staying power.

https://github.com/naftaliharris/tauthon is one candidate. PSF controls the Python trademark, so forks can't actually call themselves Python 2.8.

I don't expect that because the Python 2 die-hards are, shall we say, not a maintainer-friendly crowd. You'd have to pay me FAANG bucks to even consider it.

Perhaps, but only because the PSF wields the Python trademark to prevent people who want to continue releasing the Python 2 language from doing so:

https://github.com/naftaliharris/tauthon

Rightfully so - it would be incredibly confusing to have a third-party python with the same name but maintained by a different group of people. There's no "right" to the python name.

If you actually look at the issue discussing this, you can see that Guido is being more than amenable and the responses are honestly verging on childish.

Well put. That's been my feeling about all of this in a nutshell. Guido was a saint for putting up with all this as long as he did. Some people really need to get a grip.

Sure, I agree generally (which is part of why forking the language under the same name was a poor choice!). However, python.org has demonstrably zero interest in the Python 2 language and CPython runtime. So they can both say "see, there's no interest in community maintained Python 2" and also prevent that community from organizing itself under the name everyone uses at the same time.

IMO, Python 2 is a genericized trademark like Kleenex at this point.

> IMO, Python 2 is a genericized trademark like Kleenex at this point.

Kleenex is often used to refer to facial tissue, regardless of manufacturer of the facial tissue.

Python 2 is used to refer to one of the releases of the CPython 2 interpreter, most often the latest release, with earlier releases typically referred to with their first point (e.g. "2.6" as I've seen elsewhere in this thread). People rarely refer to Pypy or Jython as "Python 2", at least in my experience. They may refer to compatibility with a point release of the official CPython.

For example, Pypy describes itself as with the following:

> PyPy implements Python 2.7.13 and 3.6.9. It supports all of the core language, passing the Python 2.7 test suite and most of the 3.6 test suite.

They're not saying PyPy is Python 2, rather that it implements the programming language represented by point release Python 2.7.13. Elsewhere, they talk about compatibility with Python. Nowhere do the PyPy developers refer to their product as "Python 2".

That would be a nightmare of an experience for the developer community. No way. Absolutely do not want a hundred syntax-incompatible clones in various partial states of development.

Is the name really an obstacle to organization?

And have the Python maintainers ever claimed that there's no interest in a community-maintained Python 2?

Good! I would hate to have 300 different flavors of Python with no clear way to know which is which.

Reply on news.ycombinator.com