Not installing python 2, and just python 3, with the name python3, would have been fine.
Arch put us in a situation where it was basically impossible to run python 2 with a #! line, as some distros hadn't yet introduced a 'python2' symlink yet.
From the Arch developers point of view, the Python developers decided to release a new version of their language. They were going to support python 2 for a while, but the direction that the language was taking for the future was Python 3, so one day or another Python 3 would become the standard, so doing the switch was the good decision.
As said by @tbranyen the work of adapting shebangs have been done on the Arch side anyway, so I don't see a problem for people developing their projects in Python. Also Arch users are mostly power users (you need to understand quite a bit about linux to install it) so they are usually able to handle python 2 vs 3 errors quite well, especially as it is known than Arch use Python 3 by default.
"...however, end users should be aware that python refers to python3 on at least Arch Linux (that change is what prompted the creation of this PEP), so python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3..."
There was no guidance before the Arch devs made their choice. There's no reason to change back.
Especially since the goal is that python3 will be default eventually.
"...in preparation for an eventual change in the default version of Python, Python 2 only scripts should either be updated to be source compatible with Python 3 or else to use python2 in the shebang line."
Well, the answer is that this is a silly discussion. Arch is doing what Arch is doing because Arch is intentionally forward oriented, sometimes to the detriment of backwards compatibility with the old school (and yes, a distro not having a symlink for python2 is at this point nearly 5 years into being "old school").
Arch's change angered many in the Python community, and broke a lot of code. That Arch refuse to adopt the established convention, because it would involve admitting a mistake, shows a distinct lack of professionalism.
Well, until developers that do their work on other distributions start getting bug reports as a result of Arch's unexpected naming convention. Should someone outside of the distribution expect Arch to change so that their software works there? No. Should Arch expect outside developers to adapt their code to work on Arch? No. Honestly, I'd expect the kind of people that Arch attracts to be used to dealing with little snags in operation, and for them to just dive in and fix it on their own.
(Normally not a Windows user so I have no idea why this is the case or if by design.)
https://www.python.org/dev/peps/pep-0397/
As you say though, it doesn't rely on the names of the exes.
They both contained Python scripts, and I had to waste my time figuring out why they had broken when I had e-mails from Arch users complaining they didn't work -- and then keep replying to future Arch users with the fix.
I don't know how many hours and days of other developer's time was wasted by Arch with this change, but it certainly wasted a good day of my time over the last few years.
You seem to be complaining because other distros hadn't done the right thing. I don't see how that's Arch's fault. Are we to hold back for the lowest common denominator? Do we need every distro to join together and agree to a switchover date?
There was official guidance from Python on the switchover, this wasn't some maverick decision. It was just moving forward.
Not everyone wants to be tied to ancient stuff for backwards compatibility - if you do, it's your job to deal with that.
Because Arch did what they did, Python now recommends that Python 2 scripts start "#!/usr/bin/python2" (or the env equivalent).
> Are we to hold back for the lowest common denominator? Do we need every distro to join together and agree to a switchover date?
It's not just other distros; it's the rest of the world, including all of the scripts that people run but distros don't necessarily ship.
The right way to migrate would be to ship and use both /usr/bin/python2 and /usr/bin/python3. Leave /usr/bin/python as a symlink to python2 for a while. Eventually, drop the symlink, but do not replace it with python3. Allow users to opt-in to a compat symlink if they wish. Let stuff catch up. When the expectation that "python" is python2 has faded, then introduce a symlink from python to python3 by default, letting users opt-in earlier if they wish.
> There was official guidance from Python on the switchover, this wasn't some maverick decision.
No, there wasn't, and yes, it was some maverick decision. It was done without consultation with upstream.
(I am a non-Arch distribution developer)
The idea that we should then never use `python` to mean `python3` is just so backwards. Sure, in your corporate environment where backwards compatability is suepr important, do that.
In Arch, people have a distro that moves fast and breaks things, and we are fine with that. I'll accept the cost of occasionally having to change a shebang line (although as I have almost never had to go outside the AUR - someone else in the Arch community has almost always dealt with this for me).
> No, there wasn't, and yes, it was some maverick decision. It was done without consultation with upstream.
Fair enough, I got my timelines mixed up - but to be fair, it was then ratified as the correct way to do things because the upstream project agreed with the core idea.
> The right way to migrate would be to ship and use both /usr/bin/python2 and /usr/bin/python3. Leave /usr/bin/python as a symlink to python2 for a while. Eventually, drop the symlink, but do not replace it with python3. Allow users to opt-in to a compat symlink if they wish. Let stuff catch up. When the expectation that "python" is python2 has faded, then introduce a symlink from python to python3 by default, letting users opt-in earlier if they wish.
I don't really see why I should have to do this manually? If you don't want that behaviour, use another distro. This just feels like you feel like you should have some say in how other people set up their systems.
If you really hate it so much, don't support Arch - that'd be fair. Trying to shame the distro for doing it ignores the fact it's the core idea of the distro to do stuff like that, that's the point.
I did not present any such idea. I presented a sane migration path to where `python` means `python3`.
> ...it was then ratified as the correct way to do things because the upstream project agreed with the core idea.
No. It was because the upstream project had their hand forced and are pragmatic about these things.
> I don't really see why I should have to do this manually?
In my proposal for a sane migration path to the ideal? You wouldn't have to do it manually. The distribution would do it. As a user you'd be able to override it to speed up the migration for yourself if you chose; that's all.
Expecting that is as weird as expecting super stable LTS releases to run the latest-and-greatest of everything.
Which part of "You wouldn't have to do it manually" do you not understand?
> In my proposal for a sane migration path to the ideal? You wouldn't have to do it manually. The distribution would do it. As a user you'd be able to override it to speed up the migration for yourself if you chose; that's all.
I want to speed up the migration for everything - that's why I run Arch. That 'override' is a manual step I don't want to have to do.
So you made certain assumptions. Have you made your scripts use python2, which is safer anyway, everything would've been fine, don't blame Arch for your incorrect assumptions.
It was only because of what Arch unilaterally did that forced the community to start providing a /usr/bin/python2. Before that, /usr/bin/python2 did not commonly exist at all.
Arch is a bleeding-edge, latest software kind of distribution. They did not made the decision 'unilaterally'. They made the decision to ship the latest software for their own distribution, as they always do. It may have proven that a lot of software is extremely poor/inflexible, (i.e. rests on weak assumptions), but I am glad Arch moved forward as that showed their resolve to move technology forward even in the face of a lot of pressure from outside groups.
I just think that instead of blaming Arch, try to make your scripts more robust, i.e. loop on all 'python*' binaries in /usr/bin and use the first one whose --version gives you '2.x' instead of relying on /usr/bin/python being python 2 when the latest version is 3 and Arch is known to ship latest software.
As did all the other distributions. Shipping Python 3 is not the issue here. Breaking compatibility with all existing Python 2 scripts is what they did. The two are not mutually exclusive.
> I just think that instead of blaming Arch, try to make your scripts more robust, i.e. loop on all 'python<glob>' binaries in /usr/bin...
You want me to do that in a shebang line? No. That'd be crazy.
> ...when the latest version is 3 and Arch is known to ship latest software.
See https://en.wikipedia.org/wiki/Application_binary_interface. What Arch did was break the ABI of what /usr/bin/python means, breaking all scripts that relied on that ABI. Shipping the latest software is orthogonal to this. The rest of the world ships the latest software too, but without breaking ABI.
The ABI was never documented and at best a convention-enforced one.
I call your straw man. As I've already repeatedly said, the issue is about breaking ABI, not about what software distributions shipped, nor what they had installed by default. Making /usr/bin/python point to Python 3 has nothing to do with what distributions shipped, nor what they had installed by default. The issues are completely orthogonal. Stop trying to pretend otherwise.
> The ABI was never documented and at best a convention-enforced one.
Upstream shipped build systems that put Python 2 in /usr/bin/python, and Python 3 in /usr/bin/python3. That's about as good a definition of ABI as one gets in the free software world. Arch deliberately patched the ABI. You can try to argue that their decision to do so was correct, even though I disagree. You cannot argue that they didn't know they were changing an ABI when they patched the ABI.
But it is very much about that. Arch ships latest software, Python 3 is the latest python, python 2 is not. Ubuntu, Debian etc. ship old, outdated software regularly, Arch doesn't.
Because the latest python was python 3, /usr/bin/python pointed there. The other distributions explicitly decided not to ship the latest python.
Prior to Arch making the switch, the 'ABI' was not /usr/bin/python points to python 2, it simply pointed to the latest python release, which on most systems happened to be python 2, because they did not even ship python 3 at all at the time.
You may not agree with the decision, but to pretend like it was "out of nowhere" is unhelpful.
Well, the fact that over a decade after the release of python 3 we're _still_ arguing about it shows that lot of people think we should.
from https://www.python.org/dev/peps/pep-0394/ (not yet replaced/update) "for the time being, all distributions should ensure that python refers to the same target as python2 ."
The reason we don't need a 'bash4' is that, by and large, bash tried very hard to make sure they didn't break any bash3 scripts, so there is no need for old bash3 scripts to know they are now running in bash4, unlike python3.
However, I can predict with near certaintly, if python 4 doesn't break python 3 scripts, people will complain if "python3" doesn't invoke python 4, because they've always used "python3" to run the new python and "python" to run python 2.
Because Bash was made by, and is maintained by, people who understand how backwards compatability in their program is important.
Personally, I'd rather just use `/usr/bin/env python2` to invoke the appropriate interpreter (if you don't need to pass arguments to it), and if the symlink doesn't exist, the onus is on the user to create it. This is ESPECIALLY true for older distros.
Plus, for a sufficiently complicated application, why would someone want to pollute the global site-packages anyway? I'd rather manage most things inside a virtualenv or similar (pick your Python version), because I've seen far too many install_requires/requirements locking to a specific library version.
While this is a small issue, this original thread used it as an example of how Arch is good and pragmatic. To me it seemed like the opposite -- it broke lots of code and packages for no good reason. Why should users of older distros have to add a symlink, when before Arch everyone could be sure that '/usr/bin/python' if present would be python2, and '/usr/bin/python3', if present, would be python3?
Now I agree that using virtualenv or similar is a good idea, but this actually caused the most problems for little 20 line python scripts, because often the users of those didn't even realise they were using python.
I disagree. I don't believe it was as painful as some make it out to be. The official packages were updated within a relatively short period of time, and while the AUR packages took longer they're also not officially supported. There was also a news announcement [1]. Honestly, I think of a few other transitions over my years of using Arch as being far, far more painful. The Python2 -> Python3 defaults change isn't one of them.
Although I would agree that Gentoo's approach was somewhat better via eselect, which I believe predated Arch's migration.
> Why should users of older distros have to add a symlink
You're right. That should be the responsibility of the package maintainer. Conversely, why should I, as a developer, have to continue assuming `/usr/bin/python` points to any particular version of Python when most distributions have migrated away from this anachronism? sed does magic, and you really should be using the appropriate symlink for your desired version anyway. Python 3 isn't new. I have a hard time seeing this as problematic because 1) things change and 2) the solution is easy. I'd be happy to entertain a use case where this actually has presented material difficulties, however.
> but this actually caused the most problems for little 20 line python scripts, because often the users of those didn't even realise they were using python.
Well, I do agree: It's a problem for the end user, but the solutions aren't difficult (if someone's using Linux, they probably know enough to fix it). So, either change the hashbang, add a symlink, and--failing that--complain to the package maintainer if it's been packaged up by the distro and the upstream software itself has changed (it should).
My recommendation for developers would be to use the appropriate /usr/bin/python* symlink. Nearly all distros have them in place now (including Ubuntu from a couple years ago). Perhaps assuming /usr/bin/python will always point to a fixed version is a matter of misplaced assumptions.
There was good reason, to move to the latest version of a particular software, since that's kind of the whole philosophy of Arch, in fact the reason I originally switched to it in 2011 was because I had enough of me not being able to use the latest software on Ubuntu.
> Why should users of older distros have to add a symlink, when before Arch everyone could be sure that '/usr/bin/python' if present would be python2
If you made no effort to detect the actual version, it's wrong to assume that. It could've just as well been python 1.
It's funny. My reason for switching to Arch was almost the opposite. I grew tired of having to rebuild the latest software in Gentoo every time I updated and ran into someone on Slashdot who encouraged me to give Arch a try. Sure, there's more prebuilt packages, mirrors, etc for Gentoo these days, but then you're in the same boat as everyone else (waiting for the package to be built).
Global site-packages is all fun and games until someone loses an eye (or a dependency).
I had a fun learning experience with a large Python package once while trying to mix and match dependency versions, simultaneously trying to avoid clobbering the ones already installed by pacman. Without a virtualenv, it quickly becomes impossible to guarantee you won't break something in the process of installing something else. I never made that mistake again.
As an aside first: I sometimes make a PKGBUILD for most things just on the merit that it's better for pacman to manage site-packages on its own--or anything, really. It's a terrible idea to stuff things into global library directories that aren't managed in general (although incautious use of things like npm or pip can certainly do that for you--still an awful idea).
However, I do recommend installing packages into a virtualenv for anything with modest complexity, because some applications have requirements/install_requires that lock specific versions which may not be mutually compatible with others, and there's also the circumstance where official packages aren't new enough but you still want the package manager to maintain the dependent package. I guess you could go with versioned packages in this case, but since the official ones will inevitably find themselves updated at some point, I see it as easier to circumvent the issue in the first place via a virtualenv.
Anyway, I did the above polluting nonsense once when building a package with quite a few dependencies: Trying to maintain a half dozen individual PKGBUILDs just for dependencies and pestering easily a half dozen other maintainers to update their respective PKGBUILDs versus modifying the application's setup.py (either with patch or sed) to match older versions were both terrible ideas and scaled poorly. It's far easier to simply build what's needed in a virtualenv and avoid polluting the global site-packages even with the package manager handling them on its own. This is especially true for complex applications with many dependencies that may already be installed (but with conflicting versions).
Sorry for the rant, you just reminded me and tickled all the right spots! :)
I get your frustration though, it's a change that breaks a whole lot of existing scripts.
Also, the python2 symlink should have already been there. Bad packaging on some distros resulted in bad practices packaging by some devs. Don't blame that on Arch.
> Also, the python2 symlink should have already been there.
Nope. There was no such convention or specification at the time. There was, however, convention and specification (in the form of "what Python upstream build systems do") on what /usr/bin/python means, which is "Python 2".
There was no formal specification, but there was a convention that many distributions were already following (and quite a few others were not).
But, don't make /usr/bin/python run python3, that just confuses programs which have made the (reasonable based on past experience, and official python advice) assumption that it /usr/bin/python will run python2.
It seems to me that the same approach could work for Python.
You don't need magical file-specific juju or a wrapper script on most systems these days to select the Python version. Just call python2 or python3 as appropriate; the symlinks usually exist depending on packager, distro, etc. The problem up-thread appears to be with calling the default `python` on $PATH. Sometimes it points to Python 2.x, sometimes it points to Python 3.x. Sometimes it might even change (Gentoo's eselect, IIRC). I don't see the problem being explicit with which version is needed in the script's hashbang. Python 3 isn't exactly new, and I remember when Gentoo and Arch both migrated to versioned symlinks. It wasn't that painful.