NumPy 1.16 is the last release to support Python 2.7 | Hacker News Reader

NumPy 1.16 is the last release to support Python 2.7

github.com

240 pointshtfy96109 comments

Just rip the band-aid off. Some of the async stuff in Python 3 is awesome.

A bit curious that 3.4 has been dropped, as that's the stock Python 3 shipped for CentOS 7.

I generally try to leave the system python alone. Use pyenv or conda to get the python interpreters for your own code and apps.

This. I don't use anything but Anaconda these days. I leave the system Python alone.

How does this compare with managing python with brew?

I love pyenv. Brew is excellent for establishing a decent baseline, but pyenv is good for "we've tested this service to run under 3.5.7 and this one to run under 3.6.2, so I want to develop locally with those exact versions without them being upgraded from beneath me". And when upgrade, running "pyenv local 3.6.3" is almost all that's required to start working with the new one.

The code I've written recently relies heavily on async and data classes, both of which are recent additions. I can't imagine choosing Python 2 at this point.

My team is moving across from Bash to Python for non-trivial scripting. I hate it, but we're adopting Python 2.x. Our old servers are on SLES 11 which has Python 2.6 and even our new boxes have RHEL 7 which has Python 2.7 only by default. Getting the base system packages updated across all our systems and environments would be monumentally painful, so we're really stuck. It's not that Python 2.x is painful to use or anything, it's still a great language but being stuck using it seems pointless.

I'm really upset at the distro vendors, they should have upgraded their base system Python scripts to 3.x years ago and just included 2.7 for legacy support. RHEL 7.0 came out in 2014 for goodness sake.

> Our old servers are on SLES 11 which has Python 2.6 and even our new boxes have RHEL 7 which has Python 2.7 only by default.

Not to sound condescending... But why does this single default matter that much?

I'm sure you install lot of things onto these machines which isn't there by default, right?

Why is installing Python 3 on top of all those other non-default packages such a problem?

I prefer to install as few 3rd party packages as possible. In RHEL7 until recently Python 3 was only available via 3rd party repos, until EPEL added it. So now it's not that big of a deal to install python3, but before that why would I go about adding ius or something to get a modern python when there's a perfectly fine Python 2.7 built into the system for running sysadmin jobs?

> perfectly fine Python 2.7

Not sure how you see a system being deprecated in 2020 as perfectly fine.

Please enable additional repos (EPEL, IUS) providing sane Python versions. Python 2.6 isn't even supported by pip anymore. RHEL appeals to "enterprise" people for claiming to be able to run the same software for ten years without adaptations. It's a seductive trap imho.

why would I add 3rd party repos to install a modern python when I can just run my sysadmin python scripts under the system python without worrying about it? python2 is a perfectly good python for 99% of the things need to do

But definitely not Python 2.6!

Let's check the package "python" in Ubuntu...

14.04 (2014): Python 2

16.04 (2016): Python 2

18.04 (this is obvious but... less than 12 months ago!): you guessed it, Python 2 (!!!)

So, still in 2018 and until 2020 (when I guess the next LTS release of Ubuntu is scheduled) all instructions over the internet telling you to run "python" or "pip install" will run Python 2 if copied into your Ubuntu terminal.

I already said this in a previous discussion about Python 2 vs. 3: what a lost opportunity.

You should assume that python means “2” and python3 what it says in the name. The bigger problem is that invoking “python3” and having a script developed for “python3” still doesn’t guarantee that the script will work, there were incompatible changes during python3 history.

This has always been true. New syntax or libraries were added in almost every major Python version.

If python2 was python2.2, it wouldn't support `with`. It's just that python2 has been Python 2.7 for a decade, so it's been relatively stable.

Arch Linux has the packages the other way around: the 'python' package is Python 3 while there is a python2 package and binary.

I guess Ubuntu has it that way to stop the breaking changes however

You should expect python to be Python 2 because that's the official recommendation, for compatibility: https://www.python.org/dev/peps/pep-0394/

New code has no problem explicitly requesting python3, old code needs changing to request Python 2, but if you touch it anyways it's of course safer to make that change.

In case anyone thinks Arch is being crazy here and ignored the PEP (and doesn't read the PEP), note the change here was written in 2011 and approved in 2012 based on Arch's experience moving to python 3 by default in 2010.

there were incompatible changes during python2 history as well. RHEL shipped Python 2.4 forever, and now ships 2.7. There are many breaking changes between those two versions.

#!/usr/bin/env python3

For cases like this, I compile the scripts with nuitka. It's heavy, but worth every additional MB since it avoids massive technical dept. It also makes deployment as easy as scp.

Same here. Dataclasses and type annotations are fantastic, not to mention f-strings.

This is all fine for personal/proprietary purposes, but open source can't be that cutting edge. While type annotations are probably fine (3.5), f-strings (3.6) or dataclasses (3.7) won't make it to mainstream OSS anytime soon. But they are great features nonetheless.

> open source can't be that cutting edge.

what?

I mean, RHEL 8 is shipping with Python 3.6 and Python 2.7 available. There is no standard /usr/bin/python without making the symlink manually, and the system uses its own internal python for dependent tooling, so it doesn’t conflict with user installed versions.

And, based on my experience, general open source is usually much more cutting edge than the larger distros, that’s the whole point.

> the stock Python 3 shipped for CentOS 7.

What package are you referring to? As I understand it there is no "stock" Python 3 in CentOS 7. There is just what you can get from secondary repos, like EPEL, SCL, and IUS. Those repos now have 3.6 available.

However system level stuff will install 3.4 by default.

Which system level stuff? It’s an EPEL package, only EPEL or non-RHEL repo packages will call it in. Nothing from the base/updates/extras repos will pull python34 in to my knowledge, or any package non-base/updates/extras packages for that matter.

I need to reach out to the cmake3 maintainer because they just added it as a dependency and I have no idea why.

Sphynx for the docs? Homebrew add that as well to stuff which requires cmake to build.

On a fresh virtual machine built from the 7.6.1810 minimal ISO the system Python is 2.7.5:

    [root@localhost ~]# python --version
    Python 2.7.5

RHEL 8 is where they are making Python 3 available as the system Python.

https://developers.redhat.com/blog/2018/11/14/python-in-rhel...

If you need the new numpy, get a new python. Conda makes that painless, but whichever way you get numpy, python itself is easier to build/install.

>Just rip the band-aid off. Some of the async stuff in Python 3 is awesome.

Will that awesome async stuff help us migrate several 100s of thousands of lines of 2.7 code?

I guess 10 years was more than enough time, maybe 2.x series will be the Cobol of Python world.

10 years where the first P3K versions were unusable and the rest still didn't have tons of dependencies?

Yeah, right. There's a reason even companies with tons of resources like Google still use 2.7.

12 years. It's all been very usable for 6 years. So you had 6 years to use the six or python-future lib, and 6 years to remove it. 100k lines, it's a 2 weeks job if your code is properly tested. It's quite ok given python is 25 years old and asked for it once.

But good news, you don't have to do it. Python 2.7 will still work after 2020. We, the community, will just stop working on it for free. You can start paying us, or do the job for free like we did since 1991.

You can also migrate to another tech, but they will break things as well, and they won't give you 12 years to move. Ruby and node gave 2 years. Node forked 2 times. And it's a google based tech.

I'm tired of hearing complains, given how good you have it.

>12 years. It's all been very usable for 6 years. So you had 6 years to use the six or python-future lib, and 6 years to remove it. 100k lines, it's a 2 weeks job if your code is properly tested

It took Flask several years, complaints, and a custom change for their use case, to port over.

Some people have actual production code which they don't rewrite for the fun of it (and even less with comically underestimated "2 weeks per 100K" runs), because the core decided a non-backwards compatible version is the future.

>But good news, you don't have to do it. Python 2.7 will still work after 2020. We, the community, will just stop working on it for free. You can start paying us, or do the job for free like we did since 1991.

Well, you're not the "community", at best you're one committer. The community (not necessarily the core devs) will fork and maintain way beyond 2020, and it will be for free too.

>I'm tired of hearing complains, given how good you have it.

Sorry, didn't know people must walk on tiptoes lest they tire you with their complains...

Well, programming languages evolve and to maintain a clean language (which python aspires to be), it has to cut some ties.

I think the overly backward-compatible way in which the change was managed really prolonged the adtoption and getting python 3.x to mature. The Python-Community was hanging between those versions far too long and it really hurt the ecosystem and the whole python-experience.

> Some people have actual production code which they don't rewrite for the fun of it (and even less with comically underestimated "2 weeks per 100K" runs), because the core decided a non-backwards compatible version is the future.

Of course, but you (probably) still have to maintain the code. And it's not a full rewrite, more an adaption. I find most changes to be quite mechanical. Also, it's not the python-comunities fault, you chose python and you're getting probably using it for some business purpose. So i think the python community can expect you to maintain it, including language adaptions. You can still ignore this and don't update, but libraries will probably start getting incompatible and there will come a time when things break. Wheter that's important enought to migrat is up to you.

I don't think it's fair to just complain loudly, after all, the python community is not expection a full rewrite. They have given enough time to migrate, i think took much, but it's still a lot. Being completely backward compatible forever was never promised, and i think is an complete anti-feature. Write once and never think about it is not the python-way, but having a clean, smart, elegant language. It also really hurts java, which just walks with all the backage of the past and glacial speed for language-updates, which probably is required when everything you do is going to be kept forever in the language.

Says the person who has benefited from it for free for 25 years and doesn't have to pay the price for maintening it even further.

I'm a big fan of Tauthon, a backwards-compatible Python 2.7 fork with some features backported from Python 3. Personally I've yet to see a Python 3 feature that would be useful in my code.

After 15 years of python, going from client to client, my experience is the opposite: i've yet to see a code base that would not benefit from python 3. Usually people saying that just don't know much about python 3.

However, I understand that those benefits might not be motivation enough to migrate in some contexts.

In those cases, I advise to freeze the project in time, isolating it by vendoring dependancies and either compile it with nuitka or distribute it in docker.

The issue was that some people didn't even start writing Python 3 compatible code.

I would go so far as to say that Python 3.4 was the first one you could really use in production, that leaves you with 5 years to handle the upgrade. Still before that you should at least have started to add basic stuff like importing print_function and at testing all new stuff against Python 3. Many just kept ignoring Python 3 and wrote line after line of new incompatible Python 2 code.

I no longer feel sorry of those who are stuck on Python 2.7 and now have less than a year to upgrade. At this point it's your own fault.

What percentage of those lines do you estimate will need changes?

Imagine that's part of the challenge. Since python is dynamically typed/duck typed, it's hard to audit all usages of a variable. I.e. If I see a py2 string declared, how can I be 100% sure I don't break downstream components by making it a bytes or py3 unicode string? Some might be using it as the former, others the latter.

You'd need insane test coverage, of all possible code paths, to be 100% confident.

I was responsible for the Python 2 to 3 migration for various code bases at work. I used 2to3 to check for issues, but did most of the code rewrites by hand.

Most of the conversion difficulties were related to Unicode/text/binary string handling in Python 3, and 2to3 didn't catch all of them. This was the biggest challenge of the entire process. Stuff would fail in production because due to improper string handling. Python 2 was remarkably permissive (i.e. loosy goosey), whereas Python 3 is stricter and arguably more correct, but this strictness has a cost.

The other class of problems is the restructure of certain std libs, like urllib. We took the opportunity to move away from "urllib" to "requests".

This site [1] was invaluable in understanding the migration issues.

All in all, apart from breaking Unicode issues, the migration process was fairly easy.

[1] http://python3porting.com/problems.html

The problem is which lines. I found code all over the place which broke for unicode reasons (we were reading in the output of GCC, and printing it back out, and it didn't work any more as GCC doesn't output valid UTF8 in some cases).

In the end I decided moving to Rust was less work than Python 3 (in terms of being sure my program was reliable and wouldn't error out with unicode errors).

> I found code all over the place which broke for unicode reasons (we were reading in the output of GCC, and printing it back out, and it didn't work any more as GCC doesn't output valid UTF8 in some cases).

That means the code was working with raw bytes, not utf-8 strings, so that's what you should convert to on Python3.

That means using `.encode()` and `.decode()` or using bytestrings.

Python3 doesn't break string usage, it makes you do it correctly. Expect a lot more of that kind of pain by moving to Rust (not saying it's not worth, you'll indeed get more correct programs done).

The problem I had with python is that (because it is dynamically typed), and quite a few methods take both bytes and strings, it can be tricky to track what you have and often things don't break until you feed in a non-UTF8 string.

I found it much easier in Rust, as the two types are just distinct, and an incorrect program just fails to compile.

In Python 3 they are two distinct types as well.

They're not really "types" when you only find out at runtime.

When handling tricky string issues, my first step is usually to explicitly type annotate the code (using mypy annotations). This makes string handling and tracking much simpler.

Sure they are.

100% will need to be tested.

Anything dealing with text for starters.

3.4 will reach EOL very soon.

> The expectation is that Python 3.4.10 will be released in March of 2019, and this will be the final release of Python 3.4.

https://www.python.org/dev/peps/pep-0429/#release-schedule

async stuff is a bit suspicious to me. I use an async web server to command an async websocket client and sometimes you get hard to debug problems like throwing an exception in one library causing the other to stop working. That's never happened to me in node.

Debian also does not have anything above python 3.4 on the stable packages branch

3.6 has been dev and test since 2016

Debian stable (Stretch aka Debian 9, released summer 2017, regular support through ~summer 2020) has Python 3.5. Debian oldstable (Jessie aka Debian 8, released summer 2015, regular support through summer 2018) had Python 3.4. If you're on oldstable it's time to think about upgrading....

The next stable release (Buster aka Debian 10) is just starting the freeze process now and expected to come out this summer - Debian is in practice on a ~two-year release cycle, and regular support for each release lasts one year after the next stable release comes out. Buster will have Python 3.7 as default.

wow 3.5

I already have apps that won't install unless 3.6 is there

Sure, and those apps didn't exist in late 2016/early 2017 when Debian stable was being finalized. It's called stable for a reason. If you want a faster-moving distro with correspondingly higher risk, there are multiple excellent choices, but in general you'll have to keep up with a much faster upgrade cycle. (My personal preferences would be Ubuntu non-LTS or even Debian testing, which goes through pretty good QA these days and is a fine choice for non-production-critical machines.) Alternatively, wait a few months and the next stable release will have you covered.

Why in the world is Debian like this? Their buggy latexmk is also from 2015.

Somehow I feel like every system that I want to keep updated stays on an old version, and every system that I want to keep pinned to some version just force-feeds me its updates...

Because it's the stable branch, and it gets no major version updates for anything after release, nor any new packages, just bugfixes to the current version.

There is backports, and I've thought about maintaining a newer Python interpreter in backports (when I was working for a company using Jessie and trying to use a Debian stack for everything and sort of wanting async syntax). You could introduce python3.6 or python3.7 as a new package in backports. But it would be extremely hard to update the Python modules in Debian to the newer version, too, because there's just a "python3-foo" package for each module, so you'd either have to recompile all binary modules and patch them to work on the newer Python as necessary (and patch anything using async or await as regular identifiers....), or you'd have to introduce an interpreter that intentionally wasn't compatible with modules installed as Debian packages. (Or you could introduce an interpreter that tried its best and failed randomly, which seems worse than either option.) I'm personally sympathetic to wanting to get /usr/bin/python3.6 from my OS even if the only useful thing to do is put a virtualenv on top, but I think it makes sense to just install Python yourself into /usr/local at that point.

Re latexmk, it looks like the volunteer maintainer hasn't updated it since 2015, and there's a culture of not stepping on people's toes and waiting a while before assuming they're inactive or no longer care. But it's been updated in the development release. The current changelog https://metadata.ftp-master.debian.org/changelogs/main/l/lat... has a bunch of "NMU" entries, non-maintainer uploads.

> Because it's the stable branch, and it gets no major version updates for anything after release, nor any new packages, just bugfixes to the current version.

No, Debian stable doesn't get bugfixes, only if those bugfixes are for security issues! (or some really severe bugs)

Example: Debian's Python is still 3.4.2, not 3.4.9!

Thanks for the explanation.

Re latexmk: so am I getting the development release on Arch? I get the 2018 version with texlive-core.

Sorry, I mean that the volunteer maintainer of the latexmk package in Debian hasn't updated it since 2015. Arch probably has a newer version from the latexmk authors (and I think Arch does not have the same culture of individual package maintainers, in general).

Just to note: in Arch we almost only have individual package maintainers.

..Oh. Wow. Do distros not care if their packages are updated for 3+ years? I'd have expected half their entire job is to make sure package maintainers are actually maintaining packages...

Eventually such packages get removed (usually when they stop building for whatever reason). But if the package still works, having an old version of it is still better than not having anything.

I don't know who these "distros" people are. Humans care. But no humans cared-enough to update this earlier. If you want to contribute to the Debian project specifically, talk to me.

The Debian LaTeX maintainer was essentially harassed out of the project by the Anti-Harassment Team.

This is 1) irrelevant, he was not the maintainer for latexmk (in fact he did some non-maintainer uploads very recently for it to update it to the latest version), and 2) not true, he was not "harassed out of the project," he was removed by formal action. I read "harassed out" as "pressured into quitting" or something. He didn't quit under pressure; his account was removed. (And he's still a Debian Maintainer, same status as me.)

> he was not "harassed out of the project," he was removed by formal action.

Ah, yes, the standard bureaucratic cover for abuse: retroactively justify it with formal committee action, by asking people to submit examples of badthink and badspeak from years ago:

		2012: 1 email
		2013: 7 emails
		2014: 5 emails
		2015: 1 email
		2016: 0 emails
		2017: 2 emails
		2018: 2 emails

Nothing wrong with that at all. Nothing to see here! Debian's a perfectly healthy project. Continue on your way.

Source?

The horse's mouth: https://lists.debian.org/debian-project/2018/12/msg00032.htm...

Downvoters: If you're honest, then do your homework instead of downvoting. If you're dishonest, and you're trying to cover it up, shame on you--you're as bad as the abusers in Debian.

Debian was never cutting edge. If you want that, you would either have to switch your branches or change the distribution. There are quite a number of other distributions based on Debian with more recent software.

But I wouldn't want Debian to change their release behavior. And to be honest regarding different versions: Python has always been a special needs kid in this regard.

I feel like there should be a more reasonable middle ground between 3+-year-old software and cutting-edge?

True, there probably is, but Debian is pretty known to be conservative towards newer releases.

Is there any distribution besides Arch that isn't like this? Alpine is also pretty bad about this, and I haven't tried other distros in years.

No idea. I use mainly Debian but I think Ubuntu has a more aggressive update policy and is based on Debian. Not sure about RH since I don't know a lot about their package managers. But CentOS and Fedora are probably also quite up to date.

Oh Ubuntu is like Debian to me... in fact when I was taking about Debian this whole time I was actually talking about Ubuntu. Its latexmk is the one that's old.

Most of the big maintaining teams (programming languages and DEs) are the same for both distributions.

I find Nix is pretty up to date, and helpfully you can use it on top of any distro.

> Somehow I feel like every system that I want to keep updated stays on an old version, and every system that I want to keep pinned to some version just force-feeds me its updates...

Spot on.

It's not very clear, but this release is last release supporting 2.7 & 3.4. 1.5.6 supports 3.4.

I don't understand your comment. Version 1.5.6 of what?

The NumPy 1.16.0 notes say

> Support for Python 3.4 been dropped, the supported Python versions are 2.7 and 3.5-3.7.

That definitely sounds like NumPy 1.16.0 doesn't support python 3.4.

When can we start calling Python 2.7 legacy code?

Now.

seriously. I think we have to treat it as legacy code now.

2009.

Seriously, it’s been 10 years since Python 3 came out, Python 2.x has been supporting legacy code since then.

Python 3.3 was the first version >3 that was actually "good". Prior versions were, if not unusable, not a great experience. Even when writing new code. And some of the Python core devs (Raymond Hettinger) share that opinion.

And let's not understate what a big deal it was to break compatibility like that. Porting a sufficiently complex project is non-trivial.

Biggest failure of the Python group is definitely the 2.7 / 3.x nonsense. It annoys me greatly when I see new code on 2.7, thanks a lot Nordic.

This is the only way how Python2 should be treated.

EDIT: My comment is grossly misunderstood and I cannot delete my comment. I did not imply python 3 causes bad code, I was trying to tell a story about how "our" python 2 code is handsome and python 3 code is ugly; sorta implying python2 is not necessarily legacy code. I have absolutely no problem with python 3 and prefer it over python 2. I do not want to start a 2 vs 3 flamewar please just ignore this comment.

===

My company has >1 million line of python 2.7. About 1.5 year ago we managed to pass our test harness (about 80% line coverage) in python 3. At the moment we can't do that but we believe if we tried hard for a few weeks we can do it again too. I don't know how we define "legacy code" but I code in this codebase everyday and the code is actually very pretty and understandable. We also have a python 3 project (only a few thousand line of code) and god that code is definitely ugly as shit. Even after python2 goes unmaintained, I will still think our python3 code looks more like legacy.

Plenty of old code is beautiful. "Legacy" code is code that relies on technologies that are no longer supported. That's not to say I don't sympathize and I know that the transition is not trivial in a large codebase, but when it comes down to it yes, it's probably necessary to start thinking of Python 2 code as legacy.

If your Python 2 code is clean and your Python 3 is ugly, I don't know what the hell you are doing. They are almost the same language. Are you trolling?

Probably the Python 3 code was rushed and/or written by an inexperienced programmer. That would just be a coincidence, nothing to do with Python 2 or 3.

The narcissism of small differences: it makes us notice differences (and dislike the differences) between things that are mostly identical to outside observers.

> that [python3] code is definitely ugly as shit

Can you please give some examples? I've recently picked up Python 3, wondering what well known traps I should avoid.

No I did not mean python 3 is what makes the code ugly. I just said the code is ugly and it happened to be python 3. As an answer to your question, just stick to pythonic principles and you'll be good.

Python3 was cleaned up pretty well, however it's up to devs whether to use the latest cryptic features no one understands.

The only thing keeping me on Python 2.7 is Apache Beam, but it's becoming increasingly painful and I'm close to just using a different technology rather than deal with making all my code py2 compatible in 2019...

For those interested, you can track Beam Python 3 support progress here:

https://issues.apache.org/jira/browse/BEAM-1251

Right now I don't think that python2.7 is very well supported in 1.16. Had to build a docker the other day that uses python2.7 and while building the docker it got numpy 1.16 from the pip mirror. Now import of numpy failed so that I had to revert to the previous version of numpy.

"Numpy 1.16 is good enough" - Me in 2029

Clinging on to the past for dear life..

Or, if it ain't broken, don't fix it.

Well, it’s open-source so you’re free to volunteer your resources and maintain it yourself

Well, it's dependent upon by lots of huge companies, so I'm sure it will forked and maintained anyway well beyond 2020...

Strongly wishing things to die doesn't make it so...

Great, so everyone’s happy!

Reply on news.ycombinator.com