Python 2 removed from Debian
bugs.debian.org
bugs.debian.org
I have this opinion about no other language. I am generally neutral about languages I don't use. But literally my only experience with Python is porting to Python 3. It's like a sitcom scenario of how to force the worst possible first impression for a language ever. The amount of slowdown and pain that Python 3 has generated on our non-Python project is really just out of this world ridiculous.
The only reason I can come up with is that the installation process is wrongly changing the `Python` symlink. Anyone have other known causes? Does it screw with dependencies?
The worst part is a lot of times, since Python 3 is a breaking change, people decide that it's a great time to finally do all their breaking API changes too, so all of a sudden updating something to the Python 3 version means that all of its associated config files have changed format, and now you're updating config files that have worked great for ages and running into bugs there.
Your issue with Python is that some people having bad development practices?
However, I will take your snarky reply and turn it into lemonade by using it as an opportunity to talk about how often development teams fail to consider the social implications of their changes. As is mentioned in another comment on this post, the Python team seems to readily admit in retrospect that they didn't predict the pain of having no clear transition path, etc. I think another lesson to take from this experience is the realization that if you create some big "line in the sand" update, you may absolutely inadvertently encourage bad development practices. If the Python 2-to-3 transition had held "ease of transition" as a core value, then it is more likely that that value would have been modeled downstream. By instead initially declaring that the 2-to-3 transition was going to be some watershed moment and people were going to have to update everything, it is not surprising that it could encourage people to just pull the trigger on a bunch of breaking stuff they wanted to do, and come to the (incorrect, but understandable) conclusion that they didn't themselves need backwards compatibility since the entire ecosystem was going through a "unique" non-backwards-compatible change anyways.
Most of the major Python packages did maintain 2 and 3 compatibility at the same time. Maybe a few packages had bad coding practices but you’re going to hit that in any language.
I think you’re extrapolating unfairly
But that's exactly the nature of doing this sort of change. Again, I am not a Python person. This transition created an unintentional filter where the utilities with poor practices surfaced to end users, such that their only direct interactions with Python are always negative. So sure, the extrapolation may very well be unfair, but that is the completely avoidable situation they set themselves up for. No other language expects end-users of the programs written in that language to be sympathetic to some internal engineering transition. All this stuff may have great reasons, but you've already failed if you have to explain those reasons to non-Python users, or tell them this is confirmation bias or whatever. It's just not happening with other languages. And this is for big stuff too. Ansible, which uses 2 pythons, one locally and one remotely, was a pain. Add on to this that this was a fairly public boondoggle, that the makers agree wasn't handled well, so whether "fair" or not, this is a common take away I think.
Whether that’s libc requirements for compiled languages, or Perl language versions being breaking, or even as subtle as differences in behaviour between different compilers such that your system may give different results than another user.
So I disagree with your assertion here.
Yes it’s unfortunate that this is lifted up to the user, especially because of the fact it’s interpreted, but it’s not anywhere near unique to Python.
Therefore I think it’s not a logical outcome to outright avoid Python.
That does not dispute the original claim though. It's still a problem for Python even when other languages have the same problem.
> or Perl language versions being breaking
Perl 5.0 code written in 1999 will still happily run in 2023. In part due to the fact that breaking changes are *opt-in* within a Perl script.
Reference: https://www.perl.com/article/announcing-perl-7/
"""Perl 7.0 is going to be v5.32 but with different, saner, more modern defaults. [..] Perl 5 still has Perl 5’s extreme backward compatibility behavior, but Perl 7 gets modern practice with minimal historical baggage. """
Honestly, Perl is an example of extreme backwards compatibility that has the user experience at the forefront of their design decisions.
I’m saying that avoiding Python because of it is illogical (to me) because the problems they’re describing aren’t unique to Python, and I don’t think there’s much that doesn’t suffer from it.
Perl is an odd one because, as your comment says, Perl is going 5->7 because 6 was such a disastrous break.
ack
> I’m saying that avoiding Python because of it is illogical (to me) because the problems they’re describing aren’t unique to Python, and I don’t think there’s much that doesn’t suffer from it.
I don't completely agree, but I see your point. Any interpreted language (or non-statically linked binaries) could (and does) suffer from these problems. I'd like to see Python have a better packaging story...but for now we live with shipping the developers environment (via docker or virtualenv) for running complex Python applications and services.
I will say that *statically linked* compiled languages are able to avoid many of these problems because running these binaries typically only requires the OS (eg, Go, Nim, Rust, Zig, C/C++ when statically linked, Object-Pascal, etc)
> Perl is going 5->7 because 6 was such a disastrous break.
Agree. And they did the right thing by actually renaming it Raku and skipping the version release.
should be >Agree. And they did the right thing by actually renaming it Raku and functionally killing the language.
It was to precisely avoid this scenario where you get it switched out from under you.
This doesn’t mean every distro followed suite (though most popular ones were good about it) , but the biggest issue is users. Tons of users create the symlink or alias for convenience and then forget about it till stuff breaks.
> Distributors may choose to set the behavior of the python
> Avoiding breakage of such third party scripts was the key reason this PEP *used to* recommend that python continue to refer to python2.
Emphasis mine.
You won't know what version of Python 3 you're getting, but at this point it's safe to assume you're getting a version of Python 3 when you call python.
Only old scripts written when Python 2 and 3 were maintained separately should've ever had this problem, but we're past that point now.
How about one with dataclasses support? Again, you get to install a new interpreter everywhere.
Python packaging continues to be a sore spot for the language.
I mean the Python ecosystem is such that now I have to "package" up this script to potentially tell the user with Python 3.6 that they need to download a new interpreter version (eg 3.9) to run my application (and any dependencies). This puts all the pain of running a script on every user that wants to run it.
I've said it before, but Python requires users to replicate the developers environment...and somehow this is fine?!? Statically compiled languages (eg Go, Nim, Rust, C/C++, Obj-Pascal, D, etc) ship binaries...that just run.
Obligatory xkcd: https://xkcd.com/1987/
When in doubt, add the #!/bin/env python3.10 shebang to the top of your script to ensure the version it's running on is the same as the version you're writing the script for.
If I build an application on Linux I need to specify a glibc version or a PHP version or a Java version in some way as well. I don't see what this has to do with Python. Good luck running a jar built with JDK19 on JRE8, or an executable linked against glibc2.36 on Ubuntu 18, or a Python file written for Python 3.10 on a system that comes with Python 3.6.
Statically compiled languages come with their own problems (note that C++, Go and Rust also have system requirements, I've had to deal with a Go executable that would not work in a musl environment and a Rust executable that I needed to build on Ubuntu because the compiled version from my Manjaro laptop linked a against a libc version that was too recent).
If you need dependencies, specify and install them. If your only dependency is the interpreter, document it, specify it in the shebang, or deal with it. Python 3 is completely backwards compatible in my experience, it's only the programs that stuck with Python 2 for a decade that have trouble running on modern machines in my experience.
Yes, you are starting to see the problem. This is why I recommend statically compiled binaries as a solution to this scenario, and do not recommend any of the above.
Here's a real scenario I've had in my career, when deploying a script to 350k systems. I know I have Python 3.x, but I cannot count on the fact that every (or most) clients will have even a functioning Python interpreter. So what's the solution? Ship a docker container to everyone? There are multiple ways to solve it, but I complied a statically linked binary instead of writing a Python script...and had no issues.
> If you need dependencies, specify and install them.
Here be dragons, and this is the entire reason I avoid Python when I need my code to 100% work on another machine I can't fully control. I choose statically compiled languages in these instances. In a server deployment...sure it's fine because I can control that.
> Python 3 is completely backwards compatible in my experience
I do not agree. If you write a script that uses any of the following features and try to run on $version -1 (or more), your script will fail.
* 3.6, F-strings were introduced
* 3.7, Dataclasses and introduced
* 3.8, Walrus operator (:=)
* 3.9, merge (|) and update (|=) operators added to the built-in dict class
* 3.10, structural pattern matching
* 3.11, exception Groups and except*¶
Maybe you'll say to just install a new version of Python. All that is overhead and requires more management. Ok, that's fair.
The problem is when you want to run that script on a bastion host, or a prod server, 350k machines, or a server you don't have ability to download and install external binaries? (all of these are real scenario's I've experienced).
My experience has taught me that if I can't control the machine the script runs on, I save headaches by using statically compiled languages. As always, YMMV.
However, you can ensure compatibility around this by specifying the required version in the top of your script:
import sys
minimum_version = (3,6,1)
if sys.version_info < minimum_version:
sys.exit("You need Python 3.6.1")
As long as you put this above any code requiring any specific version (this will work with Pyton 2.7, I don't have anything older to test) this will halt execution on dependency failures.
You can include these checks in your setup.py if you want to rely on Pip so users are warned on install time. You could probably also add this to some wheel trigger if you add a local/public package as a dependency in your requirements.txt.
I think you are missing the key point here: I don't write a ton of Python. I don't write any Python. Of course you think this is old news. You write a lot of Python. You're totally plugged in. You've known this is coming, this inside baseball is "the thing" for you. My point is that, believe it or not, the rest of the world does not keep up to date with whatever the Python community is doing. This is not unique to Python. Most people in dev community X are not aware of what is happening in dev community Y. What is unique to Python is that all of a sudden we were forced to become aware as the downstream apps began to break. By creating a backwards compatibility issue in an interpreted language, you necessarily "backload" the end-user pain to when the apps, which by definition are updated later than the code within the apps, begin to break as packages are removed from package managers, or things are EOL'ed, etc. This is a situation that I can honestly not remember happening with any other language. I can't remember any other circumstance, and certainly not one where it proceeded to happen repeatedly, where I've been abruptly made aware that utilityX was written in a no-longer-supported version of C/perl/whatever (when I previously didn't even know what language it was), after I had been running it for ages.
That was it. Since then I use Git everywhere even though I prefer Hg.
Since then I only use Ed even though I prefer MS Office. What was your point again?
(With a side dish of "I'm entitled to working tools.")
That is, I believe, the entire point of the parent, is it not? That this whole migration was not quarantined to the Python developer community. That it spilled over to users in totally unrelated spaces.
A saw still works after a decade on a shelf, programs should still work too because they are tools.
I can't think of any 100-year old software that is being used today (ha-ha).
I'm sure some airline reservation systems and banking systems written in COBOL in the 70s are still around, but they all have ongoing maintenance to keep them working correctly.
The commenter's complaint seems particularly worse with dynamic languages like JavaScript and Python than compiled things. Old projects written in node are often just as hard to get running or installed as something written in Python 2. The dependencies are out of date, you have to install an older version of Node.js, or the dependencies just don't exist anymore. And JS stuff usually generates a huge tree of dependencies even if they only directly rely on a handful of packages.
End of story, I think if these Python-2 tools/scripts were such a huge part of his project that they keep breaking builds, the company should be invest time in either rewriting them in their language of choice or maintaining those projects or a fork of those projects.
One of my old bosses was strict about this - no COMPANY projects should be relying on external package repositories, unapproved third-party dependencies, definitely not some github project maintained by only a few people. Yeah, I think if your company is relying on "requests" or some other massively popular project with tons of community maintenance, you don't want to fork that and it's important to use the standard versions. But if you're bringing in something that is used and maintained by only a few people - you either need to become part of the community that maintains it publicly or fork the project and maintain your own version.
This isn't NIH syndrome, it's basic protocol for keeping your builds reliable and dependency management simple.
Why should I, an end user, ever need to build WikidPad? I don't need to do so on Windows, why doesn't Linux support the same thing, shipping an executable?
When you go to the store, you're not presented with a set of tools for making a saw, you're given a complete, ready to go tool. Perhaps the very idea of users having to build things in Linux is the problem?
It is very similar to the problems with Windows, always pushing updates/upgrades that aren't necessary, and often break things. Windows machines are far less reliable on Wednesday than they are on Monday, because of Microsoft's idea of pushing patches on Tuesdays.
Truly a hacker news experience.
I feel your response is best summarized as: "I like Python. I've never had this problem and don't think it's a legitimate problem. Only $megacorp users have this issue, and their concerns aren't important."
If you dismiss the problem (as you've done) then you don't have to intellectually address it. The Python 2 -> 3 transition was hard for those that huge productive workloads in Python2, and this pain will have a long tail.
Whoever missed that is self responsible for all self harm done by ignoring that.
Same Issue with e.g. converting Vue 2 to Vue 3 or Angular.js (Version 1) and Angular (Version 2).
I really hate when some programming languages or frameworks completely remove important functions which were used by nearly every project.
Isn't this precisely the problem containers are trying to solve? You can update your system, get rid of unsupported software, and still keep the older version of Debian within your container that depends on an unsupported version of Python.
Also, at some point old OS package repositories may be deleted (or bitrot out of neglect)- at that point building containers that depend on python2 will fail. You'll be able to use images built previously, but not everyone is backing up custom images in some remote docker registry.
FROM debian
RUN do-x
RUN install python2
Older versions of debian that still support python2 will theoritically be around forever, and any codebase that absolutely needs them should always work.
(though I contend that, given that we've had 15 YEARS of warning that this was coming, such instances should be vanishingly rare and not under active development)
This right here is the part that I'm still having troubles wrapping my brain around. People still stressing about Python 2 "going away" (it's gone folks; accept it) despite the fact that there's been well beyond a decade of advance warning, and Python 2 having been officially EOL ages ago now.
And than they will start complaining that it's other peoples fault that they didn't move.
That's fine, but I wanted to clarify - you intend to not use Python, because you have tried and it's not gone well.
Python didn't come out of the woodwork and attack you, so much as you stepped into a (momentarily) troubled ecosystem.
I understand they were dependencies of other projects. By using those, they became your dependencies.
Take some ownership of your problems, even if the ecosystem provides them -- they are chosen, and can be managed.
Distributions do a decent job distributing modules...
Utilities have an annoying tendency to rely on SomeStableModule==SpecificVersionForNoReason. This compounds the mess, sadly.
Those should be routinely addressed as part of your auditing, along with the whole of the supply chain
I can't really prescribe much... this is just generic advice.
I understand the woes, it's just not as unique as this is painted
Huh? This has a strong "No, it's the children who are wrong" vibe.
But to be more fair to your point, I think this is a dismissal of the OP concern's. A troubled ecosystem should be concerning to the Python community as a whole because it threatens future adoption if not addressed.
User's don't want a troubled ecosystem and will find languages that avoid these problems...and Go, Nim, and to smaller extent Zig, Rust, D are strong alternatives for the different workloads currently provided by Python tooling.
Python has a good moat with both ML and mind-share of developers...but at one time Perl had a similar moat...
> User's don't want a troubled ecosystem and will find languages that avoid these problems
It's true, I'm very late to Python myself -- for the exact same reason. The distribution part of the ecosystem is a nightmare.
People want utilities, but they learn more than they ever cared to because of dependencies.
I'm only learning how to manage it now because work demands it.
Prior, all of my needs had fortunately been met by my Linux distribution of choice -- the Python part was abstracted away, for the most part.
Developers of these utilities could save some (shared) effort by offering package specs/finding a maintainer, but I understand why it's not common.
(I may even build/maintain it for you, to sweeten the pot)
I guess the fuel of this thing is, 'misery loves company' -- I'm dealing with it, and I can't just whine about it.
The "I don't use it and it still affects me part" was the bait -- they do use it
Not sure where you're going with this. I didn't whine to someone else to fix it. I did take ownership. Over and over. I fixed these problems, as unrelated to my main goal as they were. I then complained about it after the fact. Seems like a totally fair ordering of events to me.
Separately, I will state that Python is an environment that seems to go out of its way to make it hard to meticulously manage dependencies if you want to. There's all sorts of competing environments and package managers. I'm sure it's straightforward if that's your main environment, but it doesn't help if you're coming form the outside.
> Those should be routinely addressed as part of your auditing, along with the whole of the supply chain
This assumes that the only available option is "manage this dependency better". That's not the only option. There is also "consider using something different that is easier to manage." Most of these utilities exist in a competitive market. And if there are utilities that allow me to choose the cadence and separately make that auditing simpler, then I will choose that. That is the point of my statement. I am not choosing to go on a hunger strike or avoiding some no-other-choice tool. Quite the contrary, a lot of these tools were chosen by other members of the team at some point, either because it's what they were familiar with or some other unimportant reason, and now they have, to their own detriment, made themselves the squeaky wheel, and opened up the range of possibilities to everything from "put in the work to maintain this tool that has currently demonstrated itself to be brittle" to "maybe I should use something else". Generally speaking, it's a bad thing to only draw attention to yourself when you are doing something bad.
> can't really prescribe much... this is just generic advice.
And yet this is a specific problem.
> I understand the woes, it's just not as unique as this is painted
It very much is unique. I'm literally not experiencing this with any other language that I don't directly use. Do I experience this with the libraries I directly use in my language? Of course. I signed up for that. I know the risks I take using beta builds and have a keen sense of the current internal transitions, etc. But I absolutely do not become aware of the latest argument in the C++ language spec, or a contentious Swift proposal, or end up becoming vaguely aware of the string implementation choices of a language I don't use. That is a unique Python thing. The fact of the matter is, Python has (recently? momentarily perhaps?) proven itself to require more maintenance than other tools I use. The community can decide who's at fault for this (language devs, library devs, end users, whatever), and even whether or not it matters, but if the message to the people outside the community is to deny that this is unique or troubling, then don't be surprised if the way they choose to take ownership is to count Python as a point against a project they are considering to use.
It was very knee-jerk to the interpretation that I had: "distance has been kept from Python and yet it's been a problem"
There hasn't been that much distance, truth be told. You did the same care for Python as you did the libraries, but it cost too much.
We're in complete agreement on dependency/manager aspects of Python -- my main point is, that's the core of it, I guess.
Everyone goes through this, it's awful and annoying, and my original post is a poor response to the presentation of that
The Python 2 vs 3 thing is settled, upgrade to a recent LTS and it's done... until a random relic utility comes around. Then obviously discard it
However that's not what you're complaining about. You are complaining about is that you want others to support python2 forever, just as you may want browsers/sites to support `alert()`, the `<blink>` tag and IE 6.0 forever. However that's not how the world works.
If you wish to use something nobody else wants to support, then you need to support it yourself.
Otherwise, take a moment to understand the thing you are using and follow any instructions to upgrade. Don't put ignorance on a pedestal and then make the argument that the fruits of your ignorance are only the fault of others and not yourself.
Incorrect. You have to understand that I am not a Python user, and don't a priori care what they do with the language. I only become annoyed when something breaks on its own, or if changing one unrelated thing breaks other unrelated things. This happens for a variety of reasons:
1. Either date-based breaking, or patch releases designed just to throw up deprecations, and breaking the tools you actually use.
2. Packages and or binaries disappearing from package managers, such that even if I have a pinned version and I make no changes, an accidental cache invalidation can now break the entire image.
3. The fact that updating one utility may break others (either because you need to figure out this python symlinking issue or whatever).
4. Python causing discontinuities that don't exist in other bundled build tools. Going from buildpacks:x to buildpacks:y, to get a new feature in A, usually doesn't affect B, but in the specific case of Python, means the installed version might change and break a bunch of stuff.
5. Python 3 changes often being bundled with breaking API changes in app updates, making "upgrading to Python3" a uniquely involved challenge for each one.
If you deprioritize "fault" for a second, such that we're not deciding who's to blame here (whether it's the transition plan by the language devs, the library or app authors, the distribution handlers, or even end users like me), you have to reconcile the fact that this is a unique situation to python. I am not describing the woes of development in general. I am describing a very python-specific set of experiences for non-python users. I did not out of nowhere decide to get angry at a language I don't use. This is the result of a repeated set of circumstances that just aren't happening elsewhere.
As has been mentioned throughout this thread, it is further upsetting that a lot of this could have been avoided. The fact that there are things that could have been done differently to make the long tail effect less painful is fairly disappointing. At the end of the day, I don't need a list of solutions. I've implemented them all, don't worry. I've gone through and read the 500-thumbs-upped threads of people repeatedly running into this exact same problem, either found an easy workaround or taken the time to update the config files for a tool that rudely put itself at the front of my priorities list in order to fix the build, or found a way to remove the utility in question completely. This is not me coming on here trying to figure out how to move forward. This is me identifying a clear pattern, and sharing what I believe to be is a common conclusion: putting "python" in the "cons" column when considering a tool.
I come from the node world. There's plenty of people that do the same for node. Some of it I consider fair, others not. But I make no illusions about one fact: if we want to change that perception, it's up to us to earn that by making the experience better going forward, not by explaining to people why their experience actually isn't real. For example, if there is a simple solution, if you just were familiar with the ecosystem, then LOG IT WHEN THE ERROR HAPPENS. If the person has to go to stack overflow and see a thousand other people running into the same thing, then you've failed.
> However that's not what you're complaining about. You are complaining about is that you want others to support python2 forever, just as you may want browsers/sites to support `alert()`, the `<blink>` tag and IE 6.0 forever. However that's not how the world works.
This is also hilarious since you had to scrape the bottom of the barrel to find the only instances of "non backwards compatible" features in JS/HTML, an environment notorious for backwards compatibility. Also, `alert` may still stay in.
Avoid writing huge, long rambling comments and try to simplify your arguments to less than 100 words. You may find the fault, and the fix, becomes obvious.
Start with trying to refute "The python2 executable still functions correctly as does all other code you have that relies on python2" without referencing anything specific to your build practices.
> This is also hilarious since you had to scrape the bottom of the barrel to find the only instances of "non backwards compatible" features in JS/HTML, an environment notorious for backwards compatibility. Also, `alert` may still stay in.
Those where the first that came to mind. There are many others, and many more if you widen it to server-side JS.
If you don't want to read the comment, then also don't bother responding. The comment is long because I am explaining a death by a thousand paper cuts situation. Don't worry, Python isn't going to die just because I don't like it. You don't have to defend Python even if you don't feel like reading my comment. You can also just ignore it or come back to it when you have time to engage with it thoughtfully, instead of just asserting you are correct and having your only argument be that I can't fit my argument to the arbitrary word length you've come up with.
> Those where the first that came to mind. There are many others, and many more if you widen it to server-side JS.
You should stick to talking about Python. This is just not the case and not the culture of browser JS. It's honestly fairly disrespectful of the monumental work browser engineers have done to make that a reality. If you want to switch your argument to server-side JS (where blink and alert don't exist and you are thus leaving it completely as an exercise to the reader to come up with the similar situation), fine, but node 0.10 runs exactly the same today as it did 10 years ago. Again, the bar I am setting is low: have the unchanged thing still gettable and still runnable. I get it. When you work in a certain environment, you imagine all the problems it has must be common and all the benefits are unique. But that isn't the case with this particular problem.
Start with trying to refute "The python2 executable still functions correctly as does all other code you have that relies on python2" without referencing anything specific to your build practices.
Presumably you’re too “not caring about any of this” to just pin your image to “alpine:3.15” and have it just work exactly as before? And presumably you’re too not bothered with any of this to understand that you had to do this because you didn’t pin the image to begin with, so your builds are not reproducible?
And this is anyone else’s problem why?
My point has always been simple: Python causes more pain than other languages when you're not directly using it. I can get node 0.10 on alpine 3.17 just fine. This demonstrates that your comparisons to server JS are completely wrong. But I know, you're going to tell me this is my problem and I expect the world to do work for me or something. Nope. As I've repeatedly stated, I've taken the time to get it to work already. I know you want to believe I'm here waiting for you to fix it, but I'm not. All I've said is: "Huh, I don't have this problem with anything else. I'll keep that in mind before I decide to use stuff from this ecosystem in the future".
The argument I’m making is none of your issues stem from Python specifically. As others have said, you are conflating your own confusion and unwillingness to understand with systemic toolchain issues that do exist. Except, you are not being bitten by those - you’re still at the “shoot myself in the foot and blame the gun” stage.
What a slight of hand to say "You can get it the same way you can with node, except for you can't". You should go into politics. Anyways, you asked me for a simple example, I gave it, and now you want to say what I want doesn't count because "no one wants it". Just choose a different challenge next time.
> thus you’re complaining that nobody else is maintaining this port for you.
I'm just pointing out a discrepancy with other environments. One you initially denied existed, and once I trivially pointed out did exist, you switched to attacking the complaint itself. Look, if I'm looking at two toasters and one has less features, sure, you can bark at me and tell me to make my own toaster if I want that so bad. OK, sure. Or I'll just get the one that has the features I want. You are still allowed to believe that those features are useless, but you can't deny they exist. It is OK for you to disagree with the properties I value in an ecosystem, you don't have to resort to telling me I don't know what I'm doing just because I value a different set of properties in the ecosystem.
Someone has glued on a big red alarm clock onto the side of one of the toasters.
Your conclusion after seeing this is to write to the manufacturer of the toaster without the alarm clock glued onto it and complain that it’s lacking a big red alarm clock and say they suck because they are not copying the amazing alarm clock features that the other manufacturer has.
Ok.
Nobody is threatened by anything, and my point has nothing to do with Python.
If someone posts a comment saying “curl sucks! Every time I run it it just prints ‘command not found’” I’d of course try to tell them that’s not a fault with curl.
If they replied with “yes it is because I can’t figure out how to install it from this 3rd party site” then the discussion may become protected.
But ok, maybe we will agree to disagree.
Afterwards you’d get involved in a long weird analogy discussion about two people arguing about git. But that fault would be shared by both of them.
Python is basically in the same situation as Perl was a decade ago, with the added benefit of mainstream, living-breathing "success stories" like all the data science stuff (numpy, scikit-learn, etc), django, etc. (And Python seems to have a sufficiently large batch of pragmatic & progressive experiments that address various challenges facing users. Decades of effort went into scraping off the warts in areas like packaging, performance, stdlib ergonomics, etc. Python also successfully reformed its leadership.)
I fail to see how that is true. Python was already catching on in a big way back during python 2.7 and I'd guess it probably lost at least 18 month worth of momentum building with python 2->3.
I'd say Pythons popularity today is at best unrelated to python 3.
languages that are actually growing in userbase/popularity are not static. this is hard to accept for many users of popular languages.
OMG, that's a parody I hope!
It's kind of a shame how this transition took more than 10 years. The good part, we learned that we don't have to go through the same again. of course I dropped python (I was like 18 years old and didn't know anything about programming), I had to learn python last year and you still can see a lot of stuff that makes the experience kind of annoying.
Anyway, I'm pretty impressed how Python still is in the first positions of most charts, I still fail to see how great the language is, but hopefully I will be able to someday soon :)
Congratulations to Debian on upgrading to Python 3!
And what surprised me most is that every time I mentioned this, there would be lots people telling me how this is a superior design because you can operate string like bytes. I mean, it of course has its upside, but I don't think it's worth it if you care even slightly beyond ASCII.
For me, the unicode handling alone was worth the time spend migrating from Python 2. That was a decade ago, to finding that the "python" command still launches a Python 2.7 interpreter in 2023 is just beyond belief. Personally I feel like they should have yanked Python 2 in Jessie (Debian 8) in 2015, more realistically in Stretch in 2017.
apt install python-is-python3
I'm sure you could do things well in Python 2 with proper combinations of encode/decode, but it wasn't obvious where you even needed those because with ascii text, things "just worked" transparently. With Python 3 it's very obvious where you need encoding/decoding because bytes != str.
In the early 2000s, I maintained an py2, wxPython app with the users having the system encoding win-1250; the effort to patch this was unbelievable. The migration to python3-style handling forced everyone to think about these issues, not just few people for which things were crashing. Even just popularizing the issue was great, until then, many maintainers of third-party libraries didn't even understand what is the problem that you want to "needlessly complicated" fix in their libs.
The problem is not the end-user invoking the command.
The problem is scrips expecting `#!/usr/bin/env python` to invoke python-2.
For example GnuRadio started supporting Python 3 with GnuRadio 3.8 released in 2019, and then you had to port all your programs using it to this version. So no, in 2017, the ecosystem was not ready.
In order to support Windows better, Python 3 introduced support for UCS-4 (or worse, UTF-16) strings (depending on a compilation setting when Python was compiled) and they had to introduce extra string types to distinguish readable strings from binary strings ("bytes").
These extra types made Python 3 a lot harder to teach (I teach 30 person classes every year).
So it's not all roses now.
In the end, I got used to it, BUT I just gave up asking encode()/decode() questions at the exams. Very few people understand it, or care enough (and I understand why--it's a ridiculous thing to have). You only need it if your OS somehow slept through the introduction of UTF-8, which is backward compatible with ASCII, resilient even if there are transfer errors and can encode all unicode characters.
Encoding problem used to be really common in UNIX (and before that, in mainframes), but with the introduction of UTF-8, all encoding problems I had vanished and never appeared again.
Even Windows 10 has an UTF-8 mode now and the Windows API functions that end in "A" can be made to use UTF-8.
Now, in a sense, Python 3 has this entire complication for no reason.
That said, Python 3 is ok to use now--and, conceptually, distinguishing byte strings from unicode strings is better (for example so that you don't accidentially print the former to the terminal). It just uses up brain cycles that you could be using for solving your actual problems.
I get it from the "pass the exam" perspective, since that's one more thing to worry about.
But from my experience in teaching others, doing the conversion between bytes and string implicitly (à la Python 2's way) hinders actual understanding of this very important concept, and it's quite harmful in further study.
Bytes should be considered as a separate, more low-evel thing, away from int/float/strings; at the very least, it should be considered as bits/hex numbers. If you want strings, you explicitly encode/decode them in a way, even if everything is UTF-8.
On top of that, "byte string" is just a confusing concept. It might works for English speaker (by "it's a ridiculous thing to have" I assume you mean that, "'english'.encode() is just b'english', why bother?"), not at all for Chinese speakers, even in UTF-8. There is no b'中文' -- only b'\xe4\xb8\xad\xe6\x96\x87' which has zero meanings in their own.
And even from an easy-to-use perspective: most people don't even work on bytes often nowadays. A more abstract "string" type is all they need, without worrying about how it works under the hood (and if they do, they need to understand how encode/decode works properly anyway).
There was no conversion. `bytes` and `str` were the same type.
http://docs.python.org/whatsnew/2.6.html#pep-3112-byte-liter... says:
> Python 2.6 adds bytes as a synonym for the str type, and it also supports the b'' notation.
I just checked in Python 2.7:
>>> bytes is str
True
>>> print("Hänsel")
Hänsel
>>> "Hänsel"
'H\xc3\xa4nsel'
The point is when bytes is str, everything works just fine in Python 2 Linux with UTF-8 locale (which are used in all modern Linux distributions). No need to have a distinction between bytes and str.
That how the rest of the OS works, too. Even a lot of Gtk, Glib and so on (for example the GNOME desktop environment) assume that you are in an UTF-8 locale for file names, for example.
> A more abstract "string" type is all they need, without worrying about how it works under the hood (and if they do, they need to understand how encode/decode works properly anyway).
Ehh, we had students write drivers for measurement apparatuses and they all used Python 2 str (without being prompted to do so). No encode or decode anywhere. Of the students, almost no one who tried Python 3 for that stayed with it (instead they were using Python 2). There was just no upside for this use case.
I agree that, long term, having a distinction str vs bytes makes sense. But then you ARE juggling things that the OS doesn't need--it's basically busywork in Linux.
I'm not trying to minimize your experience--but I don't think it would happen if you tried python2 on Linux today. Not sure it was worth it breaking compat for that.
That's not true at all. I remember all kinds of encoding errors when dealing with the FS, the network or any user input when using Linux.
Unless you're talking specifically about IDLE?
Debian had Py3 for ages. It wasn't upgrade, that happened ages ago, just removing old packages kept for compatibility
The print-as-a-statement was ugly but convenient and didn't seem like a big deal, the integer division was something that you could live with once you knew about it (and you still need to know what the current behavior is), and the new string types make code more verbose, and the way they were handled introduced countless hard to find bugs during the migration.
Core developer Brett Cannon's talk "Python 3.3: Trust Me, It's Better than 2.7" goes over most of the differences at the time (2013):
The fact that such a talk exists shows that the improvements aren't worth the switch for most users. If they were, they wouldn't need convincing.
Yes, Python 3 is (mostly) better than Python 2. No, it's not even remotely worth the amount of confusion and work it caused. If Python 3 had brought massive performance improvements, or proper support for multiple threads, then maybe it would have been. As it stands, Python's biggest weaknesses remain unaddressed.
That's reading far too much into a tongue in cheek presentation title.
Besides, good devs are typically reasonably skeptical of new and shiny.
And when you've got a large Python 2.7 codebase, and Python 3 is backwards incompatible, you'd be looking at a lot of work to upgrade, so damn straight you'd need convincing to make the investment.
Hell, look at how many Java codebases are still 1.8 because of the upgrade cost of going past that, even though modern Java has many compelling features.
And expecting Python 3 to remove the GIL is incredibly unrealistic. How much Python code in the wild implicitly depends on the behaviour of the GIL? All of it.
About as much as depended on Python 2-specific features?
Removing the GIL would be bloody fantastic, but it's far more involved than 2 to 3.
More like it shows that many people like GP are only aware of the superficial changes.
And, the latest iterations of Python3 have real usability improvements - the error message improvements alone make development a lot nicer. And the perf improvements and jit coming I think in 3.13 will really make more people consider py3 for their new projects.
Have you worked with pervasively non-ASCII text in Python 2? Like on a machine where any file might suddenly turn out to be non-ASCII (even if it’s only in a comment), not just data inside a few carefully-patrolled fences? Outside of a few well-behaved libraries (Flask), my experience was that it was utterly impossible. More than half of my time debugging straightforward code was spent chasing down and fixing UnicodeErrors, knowing that some will still remain. And that includes code that used nothing but the standard library.
Now you might think that’s not important, and though I disagree (especially when talking about “most users”) that would be fair. But it is a substantial improvement going from 2 to 3.
The print() function was absolutely not worth all the carnage it caused. They could have just kept the print statement around and introduced a printing function with a different name, for example.
The fact that they removed syntax for Python 3 meant that all kinds of scientific computing code had to be transitioned even if it didn't use any strings!
It's also not a problem if you only have ASCII strings.
The Unicode handling on the other hand took ages because string handling is not easily searchable and replacing it requires understanding what the piece of code actually does.
So you have it exactly backwards: if the only migration was the print function then it would be a quick and relatively painless process. But fixing the hard thing which was the broken str/byte model was the hard transition.
I am curious if you actually went through this transition or if your opinion on this is more based on observation of others’ work.
Of course not, you’re just going to revert back to the version of bash that doesn’t require parentheses and sit on it for as long as you possibly can.
Code exists in three states: actively maintained, done/mature, and obsolete.
Python 3 attacked mature code, forcing developers to either go back into active maintenance or to obsolete it early.
I did transition some Python 2 code to 3. But mostly it was uneconomical and risky, so the vast majority of it sat on Python 2.7 until it was no longer needed.
I feel that most of the people who had a positive experience with Python 3 were people doing webdev using Django. They were working on monolithic actively maintained projects and so transitioning to v3 was just a normal maintenance activity.
Again, you are arguing that running 2to3 and testing your code was hard. I guess that may be true. But having hunted for a ton of subtle Unicode bugs that problem was very worth fixing.
Didn't the unicode changes go on until at least Python 3.6? The issue that most C APIs did not care about their byte strings having any encoding at all seemed to cause one headache after another, especially on Posix, where everything touching the operating system could return random bytes. Reducing the scope to unicode wouldn't have sped up the transition one iota.
So personally I'm ok with Python 3 but I actually never ran into the described problems despite using Python 2 for years, so it was at least not a universal problem that needed to be solved in such a breaking way.
I didn't insinuate that there weren't problems or that I wanted to keep Python 2 - just that it was perfectly normal to use it and not run into problems. I think I only remember a handful of encoding problems with my other python 2 adventures...
There’s no need to sugar coat it.
It was incompetently managed, in a way that made the technical success of the work look like a failure.
That’s on them. They screwed up.
I lived through it; I feel no particular need to smile and nod and say “it wasn’t that bad”. It was bad.
Things are good now! Python 3 is great, it’s well supported and the people involved all learnt a lot about how important managing communities is, as well as writing good code.
So, all round, a success!
…but, if you don’t acknowledge failures in the past, you’re doomed to repeat them.
> Now you might think that’s not important
I don’t think your experience with Python 2 is not important; I had similar issues with it.
No one is seriously going back to Python 2 at this point.
However, the process is a lesson worth studying; and the take away is not “what a technical success!”
… it is: never do this.
Big scary breaking migrations are bad, they hurt your community, and leave bitterness behind them.
Even if the result is better, technically; there are other factors that need to be considered too.
No, the community is stupid and wrong. My machine learning professor (actual professor, not a teaching assistant) only used Python 2 in 2017. Imagine how many people had their opinions shaped by someone like them.
> … it is: never do this.
By that standard, we would never get things like angular 2 or Raku.
Would it be better if we renamed Python 3 to some other name and pretended it was a completely new language?
However, I am firmly in the Python 3 camp. It is a better language (all the newfangled bullshit notwithstanding).
Quite possibly yes? It doesn't run python2 code, it can't be inter-linked with python2 code, so is it really the same language? The hybrid dialect that works in both does exist but you have to have compatibility shims in a few places.
By comparison, most C compilers will let you use C89 and C17 code in the same project provided you set the compiler options, and C# provided the "netstandard" target for libraries to smooth the transition between Framework and Core runtimes.
Sometimes.
I mean, Russ basically said that about all the work people were doing on the golang package manager and did his own thing, and it worked out. Good job! Sometimes, a smart person can do something that is better when they're deeply steeped in the domain.
> No, the community is stupid.
I personally find that most communities have a lot of very clever, very thoughtful people in them, and if you listen, you can learn a lot from them.
Just because they don't agree with you, doesn't make them stupid.
I think that its basic respect to listen to your community, acknowledge them and think about what they say, even if you go your own way.
...but, if you (as a maintainer) just think they (your community of users) are just a bunch of idiots, do whatever you like. They are idiots if they hang around in a community like that.
For comments and string literals in Python code, note that Python 3 changed the default from Latin-1 to UTF-8, but you can still use -*- coding: -*- to change that. Bytes literals outside of ASCII range you’ll still have to escape. As for program input that’s possibly invalid UTF-8 but not in places you care about, you’ll need to set errors='surrogateescape' or similar explicitly (or possibly set an 8-bit encoding like Latin-1, though not if there’s a chance of UTF-8 in places you do care about).
Paths are a bit painful, yes, both because there’s nothing on Unix systems that precludes /home/alice having filenames in UTF-8 and /home/boris in KOI8-R, and because NT paths are not byte sequences at all (they are WCHAR, that is 16-byte-number, sequences, in no way required to be representable in the current CP_ANSI). Having uninterpreted byte sequences as filenames would work to solve the former issue, and it does: I think most functions in os will accept bytes paths as input and treat that as a signal that you want bytes paths as output (if any). For the latter the cleanest solution is probably an abstract Path type—and you get one, in the standard library’s pathlib. That didn’t work all that well in early Python 3 versions, but it does now. (And IIUC inspired Rust’s abstract-type solution to the problem.)
Python 3 takes this fact and runs with the stance that strings should, by default, carry their encoding with them (and that encoding is specifically, more or less, a sequence of quasi-Unicode code points with unspecified internal representation). But this is an intensely half-baked solution to the wrong problem, because you need to know much more about a string than its encoding for just about any string operation you'd want to do. Just with Unicode code point concatenation, you can run into plenty of trouble with: - strings containing control characters - strings with unpaired surrogates (which okay, shouldn't really be there, but get used for various reasons anyway) - strings containing combining characters - strings in different natural languages - strings in languages with ligatures - strings with ligature-like emoji (and this is obviously not remotely close to exhaustive.)
The first python 3 versions were not great and were missing some things. I think 3.3 didn't even had pip builtin
I ask, because I know I would have wanted that.
Also, the fact that there is a talk with such a name does not mean anything other than that someone is out there trying to clarify what the transition is about.
(There could have been a video "Windows 7: Trust me, it's better than Windows XP" and you probably still would say that it showed that improvements aren't worth the switch for most users.)
use v5.24
The Py3 approach was terrible and wasted untold amount of hours just because you had to migrate everything, you couldn't just upgrade codebase piece by piece like in case of Perl.
At the very least they should've just made new one be under .py3 and just make Python3 interpreter transpile Py2 code at runtime
The Python mess was, IMO, a typical example of bureaucracy inventing for itself new but previously unnecessary work to justify its existence, so I agree that that the decisions of how to introduce Py3 features caused (and still cause) waste of immense amount of the hours world-wide that could have been used more productively. Sad.
I think very few people had trouble remembering them, and if you're using one, you'll be looking up documentation for anything with more than a few methods.
It really is frustrating in these situations with high level languages. It's super possible to do the majority of changes they did in non breaking ways.
Hell, they do it already with the __future__ package. Where did all that compatibility work go in python3?
There was some breakage between 5.6, 5.8, and 5.10; Unicode is hard, and it takes time to get it right. But I think the key difference I've heard as a mostly python avoider is the intent was for code written for 5.6 to probably work in 5.8 and 5.10 and if it doesn't, for there to be a way to have one file that works for all versions.
From what I understand, it's not easy to have a python module/script that works in 2 and 3, and you can't go to 3 unless your dependencies do, so if you have a lot of dependencies (as is modern), you're stuck on 2. Your dependencies won't want to move to 3 either, because their users are stuck on 2, so if they just switch to 3, they're droping users; instead they need to support two parallel versions of their code. Most perl modules didn't have to do anything special to support 5.6 and 5.8, but if they did, it was usually small and it could be done within the file with conditional compilation --- I don't think that was an option for python.
.. but you can't use old libraries in new code.
I think having Python 2 vs. 3 is a clearer demarcation, myself, but the Perl community cares a LOT about backwards compatibility.
Also, Perl 5 is a very different example. While I haven't followed Perl for quite a while, ISTR that Perl had a very similar transition from 4->5 (I personally remember spending a lot of time trying to get old code working on newer OSes). Also, AFAIK but I may be wrong, Perl 6 has not really taken off like Python 3 did.
That's terrible practice.
Personally I only use python for throwaway scripts. Whatever is written in Python likely won’t work in 10 years anymore. So they’re all lost in time. Whereas I can continue some of my C++ eternity projects and they still work perfectly fine.
If you don't write special in your code, Perl doesn't have a "say" method (because it didn't have one in Perl 5.8). If you declare in the code that you're code for Perl 5.10, magically there are all the Perl 5.10 features, including "say". You can also declare you want specific features, rather than declare a specific Perl version.
$ perl -v | head -2 | tail -1
This is perl 5, version 30, subversion 3 (v5.30.3) built for x86_64-cygwin-threads-multi
$ perl -e 'say 1'
Number found where operator expected at -e line 1, near "say 1"
(Do you need to predeclare say?)
syntax error at -e line 1, near "say 1"
Execution of -e aborted due to compilation errors.
$ perl -e 'use 5.010; say 1'
1
$ perl -e 'use feature "say"; say 1'
1
If only Python were as reliable for backwards compatibility.
The Perl community is going through their own Py2->3 moment, and they're doing it with great care. They're going to create Perl 7, which will be Perl 5 where the "defaults" make the language look more like Perl 5.32 rather than Perl 5.8 [1]. Modern Perl scripts don't need so much boilerplate to enable modern features. They know exactly the trouble this will cause; if Perl 7 ever becomes the default Perl interpreter, the only one, it _will not run_ lots of very old Perl scripts untouched, because the defaults will make years worth of language changes suddenly visible to those scripts. Even if the fix is just to slap "use 5.008" on every script, they worry about it. That's a much better attitude than the Python developers.
> That's a much better attitude than the Python developers.
You do realize that this comes at a cost, right?
A lot of decisions in Python3 (and further) are based on the desire to keep the interpreter simple.
Python Core Team doesn't have unlimited resources, and even if they did, the cost of complexity grows non-linearly.
py2 -> py3 transition did not go as planned, but that doesn't mean that following Perl route would have had better outcomes.
Hell, if you're still using 3.6 you're limited in your ability to upgrade dependencies, 3.8 is generally the minimum supported Python version these days.
I tried using the library less than a year ago. I must have missed something. This would be great news… if the way you told me didn’t make me feel like shit.
Where would they go from python? Node?
It may have lost a lot of people who got frustrated by the 2->3 transition, but Python is clearly not in a bad place as a language today.
Most actively maintained Python code works with Python 3 now, but there's still a whole lot of even maintained code which uses `#!/usr/bin/env python`.
https://www.w3.org/International/articles/idn-and-iri/
https://www.rfc-editor.org/rfc/rfc6531
etc...
Just because you can get away with ignoring the non english speaking world doesn't mean Python should pretend it can.
As for RFC6531, my understanding is that virtually no email provider implements it, because of the same risks that make browsers often show IDNs as their punycode version - Unicode is extremely easy to use to confuse people maliciously or accidentally, since it contains vast amounts of duplicate characters (e.g. Latin a and Cyrillic а), even larger amounts of similar looking characters (e.g. Latin r and Cyrillic г), and even characters that are ambiguous unless you choose a locale! (the CJK problem, where the same Unicode codepoint can represent different characters based on the locale - whose locale to use when communicating between two machines being the implementer's problem).
Also, I'm not saying that Python shouldn't have had proper suport for separating byte arrays from encoded strings. I was only pointing out that there were actual legitimate use cases where a valid Python2 program was broken by the Python 3 Unicode string transition, whereas the GP was claiming the Python 2 had to have been buggy already.
Edit:reading around more, it seems that RFC6531 is getting some traction, and many providers accept sending to/receiving from internationalized emails even if they don't themselves allow you to have a Unicode email (e.g. you can't have айа@gmail.com, but you can correspond with someone having such an email at a different provider). So, email was a bad example in my list. The rest still stand.
For example, what if I want to enter "ουτοπία.δπθ.gr" in an application, via direct user input, a config file, or something else? Or display that to the user from punycode? No one expects users to convert that manually to "xn--kxae4bafwg.xn--pxaix.gr", and no one will understand that when being displayed, so any generic IDN/domain name library will have to deal with non-ASCII character sets.
The same holds true for email addresses: "ουτοπία <a@example.com>" is an email address. Sure, this may get encoded as "=?UTF-8?q?…?=" in the email header (not always, "bare" UTF-8 is becoming more common) but you still want to display that, accept it as user input, etc. People sometimes to forget that the name part of an email address is widely used and that any serious email system will have to deal with it, and non-ASCII input has been common there for a long time.
In specific applications you can often "get away" by ignoring non-ASCII input because you sometimes don't need it. For example I'm working on some domain name code right now which can because everything is guaranteed to be ASCII or "punycode ASCII", so it's all good. But in generic libraries – such as those in Python's stdlib – that's much harder.
How would you send an email to the customer support of flüge.de? How would you parse that domain?
IDN is here, and it's here to stay.
Certainly today DNS domain names, URLs, and email addresses can – and do – contain >0xff input, and for some of these that was the case in 2008 as well (URLs and email addresses – IDN didn't come until a bit later).
The Python 2 situation was untenable and lead to a great many bugs; "decode errors" were something I regularly encountered in Python programs as a user. In hindsight, the migration effort for existing codebases was understated and things could have been done better with greater attention to compatibility, but the problem it addressed was very real.
Is that true? My understanding is that DNS never supported non-ASCII and so punycode was invented.
I can count on my fingers the number of places I invoked manual DNS lookups inside production code.
At least python3 got that right, unicode by default.
I don't understand why there apparently wasn't a clearly stated preference about `python` being Python 2 and `python3` being Python 3 from the Python org.
Are there some valid reasons that made 2->3 migration insanely hard for some projects? I remember seeing blog articles whining about print vs print(), but surely there are some more important stuff.
Print was a non-issue once you could do "from __future__ import print_function". It was the unicode migration.
All string handling from I/O required at least checking.
The dependency issue was the most crippling, though: because you can't load Python 2 code from Python 3, before your project can begin migrating you have to wait for all your dependencies to update as well. This may involve them changing the types that get passed over the API from 'str' to 'bytes' or vice versa, which is potentially a breaking change.
For python2, the str class could either contain a sequence of characters, or could contain a sequence of bytes. These are often conflated when using ASCII encoded text, but are very different for any other encoding. In python3, these are represented by the str class for a sequence of characters, and by the bytes class for a sequence of bytes. It’s so much nicer to work with, but required manually separating out the earlier use case.
The whole 2 to 3 debacle was only because somebody thought that naming your string class 'unicode' doesn't sound nice enough or something.
The war continues to this day apparently with this Debian news. Somehow you can use fortran77 compilers without people getting angry but you can't use Python2
I had clean Python 2 codebases which had been Unicode-safe for years and had good test coverage, so porting to Python 3 was trivial and mostly automated. The projects which were hard were the sloppy ones which conflated bytes and Unicode, and the poor development culture which lead to that also tended to mean they had limited testing, orphaned dependencies, monkey patching, etc. which made the migration hard. I also found a fair number of cases where that process fixed other bugs which had been ignored for years.
edit: This talk he did from pycascades 2018 in particular I remember seeing and being a good retrospective: https://www.youtube.com/watch?v=Oiw23yfqQy8
The name originates from an internal script at Facebook called codemod. It had a public open source fork if you are curious.
Python’s rise brought tensions there in multiple ways - business users started wanting longer-term support and tools for migrating large code bases, but relatively few would pay for them, and some maintainers had either drifted away or retired, or simply had less time available due to things like starting families or getting promoted. The 2 to 3 migration really highlighted the gap between commercial open source and hobbyist projects, and I’m sure now anyone would fundraise for support having seen how that worked out.
The other thing we’ve been seeing is a switch to faster releases industry wide and I think that showed some serious tech debt. Projects which have CI & decent tests had a much easier time shipping updates or having two release series for an while, and the 2 to 3 migration really highlighted that. Some of the griping was really showing that older projects often had the frictional disincentive of that tool. That’s still relevant because of the way security issues are forcing more frequent updates now.
A big part of 3 was Unicode strings. In 2 you could mark a string as Unicode with a u””. This would have been a great way to let libraries and code work with both 2 and 3. They banned this, but you had to still use b””. This was the attitude and just one example.
The list of stuff that got hard / slow was long. Migration path was unnecessarily difficult as was compatible code. Somewhere around 3.4 it was like a light switch flipped and they started being more reasonable. ASCII handling improved (yes, not all internet protocol stuff is Unicode), they began making it easier to target 2/3 etc. but it was horrible to start. I’ve thankfully forgotten some of the details :)
I feel this take is outright wrong. Python provided it's 2to3 tool[1] which took care of the bulk of the work required to port Python2 code to Python3. The only code that was not supported was eggregious errors in the code that worked by coincidence, such as handling bytes as strings and vice-versa. Porting old code to Python3 is a breeze that consisted of running 2to3, run tests with python3, and if anything breaked then just touch up the code to get it to work. I know it because I personally ported half a dozen projects throughout the years.
I'll go as far as to claim that most problems porting old python code to 3 were either upstream dependencies dragging their feet or internal human/organizational issues.
Your comment is a perfect illustration of the issues. Lots of user blaming. No actual solution. As I said, it did start to get massively better at some point. Instead of condescending lectures on org issues they for example began allowing u”” in 3, which did not mess up Unicode handling in 3
I don't see where GP made a claim that it would. Why would one expect code written for a newer version, using features that don't exist in an older version, to run under the older version?
Folks using u”” were already being careful with Unicode . Supposedly Unicode was important enough for 3 to default to it.
The cost benefit of then destroying code using u”” made no sense to me. It was unnecessary. This was followed by the oh just upgrade thing. I’d love to see the “quick” upgrade guidos employer did. My guess is a total lie that it was easy and it probably took years and VC type money.
It’s not about using new features, it’s about not unnecessarily breaking old or allowing for cross compatibility.
Except they broke the old way for existing features, required you to use the new way and in most cases you could manually write code so it would still work with both. Meanwhile the only tooling provided by the python 3 crowd screwed over anyone with an existing customer base stuck on python 2, which at least early on should have been a foreseeable problem. Not to forget that anyone stuck on python 2 was literally Satan and projects where outright "shamed" to drop python 2 support.
Many folks would have been ok updating to 3 compatible approaches if it didn’t blow up their 2 story. That’s what ultimately happened in lots of cases when it became more doable. The rip and replace everything at once was a weird goal
That's what "breaking changes" and "backwards incompatible" means.
What exactly is hard to grasp?
> So you ended up w a chicken and egg problem.
You really don't. Your code and your upstream dependencies need to be ported to python3. Once your dependencies are updated, all that's missing is you doing your job.
Python2 has been on the path to deprecation for how long? A decade?
You're fabricating problems where there are none.
> Your comment is a perfect illustration of the issues. Lots of user blaming.
There is no issue. When I had to port projects to Python3, I just ported them. No drama, no hangup. You're pretending there were problems where there were none, and you're throwing a tantrum when this fact is pointed out to you.
There is really no justification for this. It's high time people like you stop making up excuses and start to own up their misconceptions, misjudgements, and mistakes.
Some projects were rewritten from scratch in entirely different tech stacks in less time the likes of you complained they could not update their projects from python2 to python3.
What is it about python 3 that brings out this lecture stuff.
I would really like to see the inside of this “easy” change at even the bigger most well resourced places. Google / Facebook / Dropbox etc.
At small places where a program is working and programmer is gone they are not investing in ports.
Anyways - they did eventually get a clue. It’s much easier NOW to handle this, but it wasn’t at the start. Yes, they did put u”” back in. No it did not destroy the world.
This saga isn't completely over, over here: We still have code running python2 that we don't have the room to upgrade, and libraries that have to work with both python2 and python3. Meanwhile, people are pushing "code consistency" to the point where people outside our team are taking those libraries and running the "black" formatter on them - but "black" doesn't understand python2, so it removes the "u" prefix and breaks the library.
I think some people convinced themselves that using "" without the u"" was disenfranchising to people who don't want to use ASCII to represent their language. This transformed a practical argument over syntax into a moral crusade.
This isn't strawmanning - this is literally the case for a codebase I'm working with today.
Though I find it hard to believe that there are some useful dependencies that have not been ported and don't have better alternatives. Typically a "never ported" dependency is never ported because it is deprecated in favor of something better.
If you're consuming dependencies which were never updated in the past decade then you have more worrying problems to deal with than porting your code to python3.
Relax things a tiny bit in 3, back port to 2 via futures, get a library like six going. It could have been a ton easier.
In fairness by 3.5 maybe they’d realized this, but a lot of lecturing was done before. Initially recommendation was to upgrade all dependencies and programs to 3 and not try and make dependencies work w both
I'm glad that this is what we have for Python now, despite the pain it caused. We'll be using Python for another few decades probably, so we'll reap the benefits.
Nah, I'm good.
The other side of that coin is that the python maintainers went out of their way to push back the Python 2 EOL. They kept on pushing the date back and back again, as per the sunset page:
"We did not want to hurt the people using Python 2. So, in 2008, we announced that we would sunset Python 2 in 2015, and asked people to upgrade before then. Some did, but many did not. So, in 2014, we extended that sunset till 2020."
The drawn out demise of Python 2 was, frankly, painful.
I have no time for whinging snowflakes complaining 12 years (2020-2008) was not enough time to migrate their code to Python 3. Hell, even the original 7 years (2015-2008) should have been long enough for 99.999999% of the community.
They had to push back the EOL because they screwed up.
I'm old enough to have read this when it came out...and it changed my view on backwards compatability (from Joel Spolsky of Trello, FogBugz, and StackOverflow fame)
"Code doesn't rust": https://www.joelonsoftware.com/2000/04/06/things-you-should-...
If a company has production Python2 application / service (with hundreds of thousands of LOC), what business value does it bring to migrate it to Python3?
At that point, if you've got to make severe changes, folks might decide to use a language that doesn't impose breaking changes (and business cost) on them. YMMV.
Not just talking ASCII was one of the first ways.
Drift in 3rd party library support is another.
Security support of the language and libraries is a massive one.
Simply put as hard as you try to stand still the rest of the world is not going to.
I feel like a lot of the people who had this reaction were reacting to being told (correctly) that how they handled strings was broken. A big reason that projects had difficulty upgrading, especially difficulty using the automatic 2->3 upgrade tooling, was because of broken handling of unicode and broken handling of binary strings. I remember reading a quick assessment of a large open-source project that there were thousands of calls to a single string-handling function that would each need to be inspected by hand to determine whether it was correct usage that would be correctly translated by the 2->3 converter or buggy usage that needed to be fixed before running the 2->3 converter, with some people guessing a 50/50 split. The effort was going to be gargantuan, and though the ultimate cause and responsibility was poor design choices in Python 2, the immediate cause was that the code was wrong, and some people didn't appreciate hearing that.
No one really blamed programmers for getting strings wrong in Python 2. It was unreasonably hard and extremely rare for people to get it right, so there was not really any shame in Python 2 programs and libraries being pervasively broken in that way. But some people still felt it was smug and condescending to make reference to this elephant in the room. This put the Python maintainers in a bad position, because they were trying to explain the benefits of upgrading and also the costs and difficulties, both of which were intimately tied to the brokenness of current codebases.
As a user and spectator to the whole drama, therefore, I felt that some maintainers got overly defensive about issues that virtually every codebase suffered from. And some of them took a "so what" attitude towards bugs involving non-Western languages, saying that if they personally didn't care whether their code handled Chinese or Bengali filenames then it wasn't a bug to do random things with them, which sounded just as bad ten years ago as it does now.
Can you be very precise about what you mean by "broken" or "incorrect" code?
Python3 enforces the difference between encoded and unencoded strings. It forces you to deal correctly with unicode. If your code base was already handling unicode correctly it wasn't much hassle to migrate. If your code base was making a mess of unicode handling --like many at the time were-- you'd run into that headfirst.
Secondly, it seems to me the Python team didn't want to make it possible to write code that works correctly on both Python 2 and 3 simultaneously. They wanted you to write code that only works on Python 3 and tell every single Python 2 user to get lost. Essentially, they wanted to co-opt you as their agent of change; burn your own reputation by abandoning your userbase (or worse, start proselytising Py3 at them) or double your own workload to maintain two codebases. Who wants that?
It's fundamentally an attitude problem. A desire for there to be only one right way to do things, a change of mind of what the right way is, and kicking the chair out from anyone doing it the old right way; the only compromise being to kicking the chair very very slowly. In Java, you can still run code that uses Vector and Hashtable from 1996, they're not even deprecated. They're used by _very_ little code since generic List and Map APIs superceded them 2004, but they're still there.
The core team was swamped with other work around the migration and the community found their response to real world problems lacking. They did things like declaring Python 2 dead long before 3 was usable. Then they refused to fix SSL problems with Python 2, which everyone desperately needed, and went out of their way to oppose a community-led release to fix those problems.
It all ended well. There was another Python 2 release, and Python 3.4 came and fixed most problems people had. It was still slower, but performance picked up and by 3.7 it was just as good if not better. But it was a good ten years during which the response to real world issues could have been a lot better.
All in all, the migration could have happened much smoother. They could have learned from other languages. But ten years is not unreasonable for backwards incompatible changes for a major programming language. The idea that the process could be accelerated by neglecting the old version does not work.
In my personal opinion, the single thing that could have made the transition better for the community would have been the possibility to run Python 2 code in Python 3. It was discussed a lot before the transition and was deemed impossible because of conflicting data types. But it's not impossible, it's just (a lot of) work. It's another type conversion, and Python can already do those. The impossibility to link to old libraries made the conversion a flag day for most code bases, and those are really hard.
You need to install either "python-is-python2" or "python-is-python3" packages to get proper system-wide symlinks. Also, it was strictly verboten to give either of two packages as dependencies to workaround migration procedures.
Debian Testing doesn't have Python2 at least for a year now.
Another mistake in Py3 migration was keeping the interpreter name as python so when you had it in path you had a choice of new stuff breaking or old stuff breaking
It was, in the past.
When python2.7 deprecated and removed in Debian testing, python ceased to exist as a command. You need to install these packages (python-is-python{2,3}) to get the python command and point to the correct interpreter you like.
This behavior is consistent in two Debian testing systems, installed 6 and 9 years ago, respectively.
> Another mistake in Py3 migration was keeping the interpreter name as python...
I installed python-as-python3 myself, after python2 disappeared from system to fix an oddball python3 script which bundled as a part of a bigger software package which is not in official repositories.
Maybe they have fixed it, but I didn't bother to check till now. I'm not even sure which program it was.
So, it was an addition to a clean system and didn't inadvertently directed python2 code to python3 interpreter.
I guess nowadays, as the py2 compatibility is a thing of the past, most programs would crash when using that interpreter version.
It will be easier for the Python core maintainers, but introducing a breaking change with the intent that that version 2 will be deprecated has probably caused more distress overall, especially maintainers of other open source projects which rely on a specific python version.
Let's never have the same kind of breaking change again, please.
It wasn't not even that easy, to be frank. 2to3 is not a small project [1] and they had (alas, incorrectly) assumed that there were enough people to have enough tests to be benefited from 2to3, only to be found massively wrong. In hindsight no one had a good idea about language evolution; the modern concensus is a single interpreter supporting multiple parallel versions, also called "editions", but (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it.
Not that I've any proof but I kind of find it hard to believe there were no other examples before
But editions & friends are for syntax. Python 3 changed semantics all around. Editions are not an option when you’re changing the langage’s entire text model, especially without static typing.
Python 3.1 could then have introduced some of the other minor changes, perhaps behind a future import.
I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version. We couldn’t have done it without six.
It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake.
Except "that's it" does a ton of work here, as the entire stdlib had to be audited and updated to correctly work with the new text model, plus other semantics change (e.g. the removal of total ordering).
That those were so large and wide-ranging in the first place is why the core team decided that fixing a bunch of other issues could go in as well e.g. reorganise the stdlib, move keywords to functions, remove deprecated stuff, etc...
Those ancillary changes were mostly pretty easy to implement.
> I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version.
It wouldn't have been more pleasant if the breaking changes had been spread over multiple versions, you'd still have eaten all of those, and the early adopter would have gotten even more pain as they'd have needed to implement breaking changes on every update.
> We couldn’t have done it without six.
We did just fine without six migrating a hundreds-kloc-scale repo from 2.7 to 3.5, we basically copied what Werkzeug did, with our own "minisix" module for just the bits we needed (some lifted from werkzeug's and others bespoke e.g. we needed a compatibility layer for CSV).
We could not, however, have done without the 3.0 - 3.3 improvements, the existence of 2.7, or the community experience.
And tests. Having a well-tested codebase was absolutely essential to a comfy migration, the less a corner of the codebase was tested, the more migration bug kept popping up afterwards.
> It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake.
No, doing all those breaking changes version after version would have been a lot worse. IME the vast, vast majority of the pain (short and long term) came from the semantics changes, not the syntactic ones. Those were relatively easy: do the syntactic change, enable the corresponding lint (so colleagues don't reintroduce the old construct out of habit), commit. A few of the API changes were frustrating (like base64 which was clearly mis-converted by the core team but what do you do).
- the text model stuff took by far the longest and had by far the longest tail (as in things we missed during the migration), this is in part because not all of the changes or Python 3 APIs are ideal, but mostly it's just a lot of runtime things in a lot of paths if your software leverages strings a lot
- total ordering removal broke a surprising amount of things, and also had a pretty long tail of bugs, mostly in that `None` would get mixed in with other values and get fed to comparison operators, sorting, or min/max
- `round` was a surprise contender for pain in the ass, not just because it changed the rounding method (to banker's rounding, from I don't remember what) but also because P2's round() always returned floats while P3 returns integers if precision is missing / None
- changes to dict iteration also broke a few things, there were multiple pieces of code with implicit reliance on iteration order, although for the most part they were issues in numerical stability of floats, which broke tests using strict equality on floats
- division changes also broke a few things as it wasn't always easy to recognise whether operands were always floats or could be int, also `//` being floor division rather than integer division led to a few behavioural changes
We also did a few things like banning the builtins which had changed behaviour (open, map, filter, ...) for the duration of the migration, the issues they caused were subtle enough as to not be worth it. They were re-allowed once we dropped P2.
Most of the other changes were kind of trivial though. print → print(), some stdlib name changes: that's easy stuff; you can automate that, even provide a compatibility shim. The division operator was a bit more tricky, but I never encountered code that broke due to it (although I'm sure some did, it didn't seem like a wide-spread issue). Plus you could do most of these things in Python 2.
> It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible.
You should have seen the initial discussions surrounding Python 3000, as it was called back then. The most common thing Guido had to say was "No, that's far too large of a change, we're not designing a new language", even for things where it was agreed on that the situation wasn't ideal. The community came up with some pretty wild proposals, and a lot was kept out of Python 3.
The difficult changes between Python 2 and Python 3 are not static information, at least not without fully typing the entire codebase, which was even less feasible in Python 2 than it is in Python 3.
https://docs.python.org/3/library/__future__.html
That only works with “syntactic” updates, not with deep semantics changes.
Those are why the core team pulled the trigger on an overhaul update. And they were by far the biggest issues during migration, anything which would be gated behind an edition or opt-in was easy.
Perl did it since ages. You just put
use v5.24
From the toolchain and language developer side, nobody liked Perl's approach. That's why there was so much effort around Perl 6 (which became Raku) at around the same time.
If you tried to run a 5.8 script on a machine without 5.8, you'd get an error as should be expected. If you ran a use 5.6 script on a 5.8/5.10 machine, I think you'd get 5.8, but I can't remember... 5.10 was less disruptive than 5.8.
I don't know where this discrepancy comes from.
(And before anyone asks, a big part of why it’s so big is that it’s very old. So old it predates Django, so, in house framework, in house ORM, etc.)
Took a calendar year of ca-a-a-arefully migrating these one by one, all while delivering stuff and making sure nothing falls apart along the way.
Not pleasant. Can't recommend.
It's been 14 years and dropping python2 is still controversial.
It's a story of how not to do a major language upgrade.
It's not quite as bad as "Perl 6" which killed off the whole language, but it's up there with massive mistakes which has cost the industry dearly.
To be fair it was a place where the Python fans advocated for its use but to management was always a bit skeptical, so we had to make it all seem easy. We had good solutions to internationalization which all broke with Python 3.
We cannot deny everything that happend before Python 3 was released.
Is there a risk it becomes like one of those situations where certain people use a topic simply to make conversation? Or the modern exaggeration to avoid seeming boring? Or a bunch of "oh, it's gonna cost ya" contractors plying the age old lines of car mechanics negotiating basically anything. Someone below referred to carnage. I wonder how far off we are to someone claiming it was actual violence.
I was lucky it didn't affect me much, but I could sympathize if somebody had hundreds of thousands of lines of python2 legacy code that needs to be migrated..
If anyone has any good resources I could browse that might help me understand the full story feel free to share.
So I started with python 2 and found the book crap. The author's idea of teaching is telling others to repeat an exercise a hundred times with all possible variations. There were more than a dozen about printing stuff to a terminal, before you got into any interesting stuff.
From their website, although they added a new course and book for python 3, they are still selling the ones for python 2. Still selling stuff for a version killed 3 years ago requires some fucking audacity.
Later I learned python 3 from other source and loved it. It's what I mostly program with today.
Starting with Java 9, finally the long deprecated APIs started to be removed.
In any case, the issue is that we are forced to write C# 7 and Java 8 code.
Some well known examples, Sitecore, SharePoint WebParts, SQL Server SP extensions, Dynamics, Office AddIns, VS plugins, Solr extensions (many places keep using Solr on Java 8 even if more recent versions exist), Android (unless one goes Kotlin or targets version 12), Forms and WPF 3rd party components (the versions on .NET Core are incompatible due to new designer)...
I am stuck on .NET48 at least for another two years (because of webforms frontend) and the last version of EF Core that supports .NET Standard 2.0 is 3.1. They dropped .NET Standard 2.0 like a rock... for what? Few default interface implementations.
Or nullability attributes that are missing in .NET Fx, they could also be defined externally, like R# does. Nope, sucks to be you.
Compared to Python 2->3, the breaking changes were more obvious as they involved entire strucutres and paradigms being deprecated. Python 2->3 felt a lot more subtle despite being just as damaging.
And if you don't want to use moin2, it can convert the custom markup to Markdown, which you can then use elsewhere.
That's my impression from https://moinmo.in/MoinMoin2.0 anyway.
If they had started a new project called COBRA and just eventually had it over take python the world would be all over it.
But massive breaking changes going from 2 to 3 was a disaster for their audience.
Worse still you could hear the acrimony in Pythons communications on the topic.
1. The amount of breaking changes. If it was possible to do a little at a time, over a series of releases, it's a bit more manageable.
2. Being an interpreted language makes it difficult/painful to break compatibility over a series of releases, as all applications and libraries have to be kept source compatible. Source compatibility also makes it a pain for library maintainers, as you can't just cross-compile the library to a older ABI.
Maybe support for consuming libraries based on an intermediate, compiled representation, would make it easier for the library eco-system to follow breaking changes over a series of releases, and in turn make it more feasible for applications to handle a small amount of breaking changes over a longer release timeline.
It was also for a time impossible (or close to impossible) to run the same Python code in 2.x and 3.x. It helped when later 2.x versions acquired more 'from __future__ import'.
They really aimed for making it maximally hard to support v2 and v3 at the same time which created a nasty coordination problem: why upgrade if the libraries aren't ready and why upgrade the libraries if there are no users.
Python source files then somehow need to announce what python runtime version they expect.
Having to support old versions forever also should make the maintainers think twice before breaking compatibility again.
The one of the things Microsoft appear to have mastery of is making sure users and devs don't feel breaking changes.
Drivers generally as well as D3D.
Even if you don't get all old code to work, having say 80% of it working means to start using Py3 you'd need to only initally rewrite 20% of the codebase, run tests and off you go, then migrate rest slowly over time.
- Added some stuff to AngularJS to make it more Angular2 compatible so that you could get closer to the target without taking in the breaking changes
- A thing called ngUpgrade let you dual-boot AngularJS and Angular2 together, so that you could transition parts of your program over time without needing to redo the whole thing
- Well documented migration guides for each step, outlining what you need to do in order to move from X version to Y version.
I suspect a lot of the controversial decusions made doing the 2to3 migration by the python foundation was deliberately designed to avoid a situation similar to what happened to perl when the time came to cleanup some of the old cruft from the language.
It was a necessary change too. Supporting Unicode strings or mixed-encoded strings was absolutely awful in Python 2 and almost not worth doing. Python 3 made that work properly.
Arguably, the only real mistake was using UTF-16 instead of UTF-8 for strings. But back then, nobody knew what would take off and Windows natively uses UTF-16, so...
Raku is a completely different language from Perl. Writing Raku instead of Perl 5 is like writing Rust instead of C++.
> Hey our interpreter for our backend code will be eventually deprecated. It is currently version 2 and we need to move to version 3"
> Hey, there is a new interpreter out there called COBRA. We need to rewrite our whole codebase to use it, there isn't a tremendous amount of benefit, but its the new thing from the same people and eventually everyone will use it. Trust me.
But it was a lot easier for the CEO to digest `2 -> 3` as a clear path to the future as opposed to switching to a new language.
Nope, the real life conversation would be "Is 2 still supported ? Yes ? Well then we will migrate when it won't"
Python doesn't exist anymore, everyone's using COBRA. We can dedicate a dozen engineer months to switch to COBRA.
Or thousands to maintain python ourselves...
Been long enough for everyone to migrate to python3, and this finally happened.
ERROR: gcloud failed to load: module 'collections' has no attribute 'Mapping'3.8 is a bigger C ABI roadblock than 3.10.
When I worked on patching/rebuilding all the Python packages for Alpine's upgrade to Python 3.10 I remember quite a few things being broken and needing patches (maybe about 5% of the overall packages), but a lot of those were also test failures due to the new deprecations, so while some stuff did truly break, it was pretty rare.
[1] https://docs.python.org/3/whatsnew/3.10.html (see the removed section especially)
I've been slowly porting a lot of my legacy code from 2 to 3 on an as-needed basis, there's a few bits though that are just too huge/crusty/etc that they will never get ported.
But it would be a mistake to consider Python anything but luxury glue no matter how much you've used it for production code, it isn't and never will be a bullet proof language, there are just too many ways in which the runtime can surprise you and the long term maintainability of any Python codebase is always going to be questionable, in part because you will most likely have dependencies on stuff that will silently disappear or change out from under you while you're not watching.
I've been writing Python code pretty much since it was first released and there isn't a single piece of code that I ever put together that worked longer than a few years out of the box without some breaking change. This as opposed to just about every other language that I've used, which I find very frustrating because Python could be just about perfect. And I'm still sore about significant whitespace ;)
The best practice, which prevents this issue, is to use envs, (as well as pinning your dependencies for any given project in a committable artifact).
(Seems like a lot of people are using system Python, which isn't really recommended.)
The whole idea that you need a multitude of environments to support applications written in the same interpreted language is where it breaks, that should not have been necessary if backwards compatibility had not been an issue.
I really like the language, but all the tooling is seriously broken.
Fwiw what I've settled on that seems the most stable is to use pyenv as the "outer layer", as it shims the python interpreter along with any venvs so you don't get conflicts / heisen-incompatibility. Lets you use anaconda envs as children as well. The real trick, I think, is to pick one system as the canonical master env controller and don't deviate.
The code typesetting is for the humans not the machines.
That is a lot worse than formatting squabbles, especially for a glue language where you are expected to be able to copy snippets for short scripts all the time.
The libraries are very powerful, but a lot of them are hacks. Numpy and Pandas totally rely on magic and overloading the array indexing operator. You can't express a problem the natural way or it will be slow, you have to think the numpy way. I find it especially confusing if you create expressions with numpy arrays. Am I operating on them element wise, or creating a "cross product", or am I creating some kind of magic indexing object? Matplotlib is also annoying, there are so many ways to do things and half of it is object oriented, while half of it uses global state.
One thing I like about Python are the web frameworks (Django, Flask). Async is also good, although it took almost 10 years for vanilla Python to catch up with what Twisted already had.
About code breaking I also have the opposite experience. I wrote a little Gtk app in the late 2000s and it worked with little or no changes for many Python and Gtk versions, thanks to the dynamism of Python. That would be completely impossible with a compiled language.
I do think that "The libraries are very powerful, but a lot of them are hacks. Numpy and Pandas totally rely on magic and overloading the array indexing operator. You can't express a problem the natural way or it will be slow, you have to think the numpy way. I find it especially confusing if you create expressions with numpy arrays. Am I operating on them element wise, or creating a "cross product", or am I creating some kind of magic indexing object? Matplotlib is also annoying, there are so many ways to do things and half of it is object oriented, while half of it uses global state." is an illustration of my point: the language itself isn't powerful enough to do the job so you rely on a lot of libraries written in different languages to glue it all together.
An 'elegant' language would provide a way to do so without all of these leaky abstractions.
To me Clojure is such a language, but it doesn't nearly have the kind of ease of use that Python has.
The language is 100% capable. Matplotlib's API is horrible and confusing because one of their design goals was/is to be similar to matlab (that's where the "mat" in it's name comes from) to allow people who are comfortable with matlab to switch more easily.
Matlab's plotting facilities are a mess of global state, so matplotlib has copied it.
> An 'elegant' language would provide a way to do so without all of these leaky abstractions.
It would have to do something else, first: Exist.
In 10+ years of programming, I've yet to ever see a implementation of a complex abstraction that was comprehensive and complete enough that I didn't find a corner case requiring me to reach through it and deal with platform specific something-or-other.
I think I like Python, because coming from C, a lot of things that are tedious there are effortless in Python (Strings! Lists and dicts, libraries, and so on.)
That's why each and every number crunching problem will eventually make use of the various escape hatches to call libraries written in a language that is performant.
As for strings: BASIC also had strings. That doesn't mean that having strings (or even dicts) is what makes a language elegant.
The problem really is that there are only so many sweet spots for programming languages and usually those come with limitations with respect to the domain you want to use them for. Pick any two: expressive, fast, easy to use.
At least not until the day they suddenly didn’t.
FWIW I found it nearly useless - whereas biting the bullet and refactoring it so that it could be unit tested - even just a few tests - made a world of difference.
That's been happening more and more recently to me. Python 2 really was lax about letting you mix types and get shit done fast, but Python 3 is a different beast entirely. Really opinionated.
Oftentimes there's... Bodges and hacks that made shit work that I never documented.
For Java 8 especially, the later reason is particularly amusing, as Oracle's extended support is now slated to go to 2030 (16yrs after release, for comparison Py2.7 got 10yrs), but Java 21, the next Java LTS that should release next year, will be on Oracle's standard 8 year extended support cycle, and thus extended support only goes til 2031. So why would anyone decide, much less how would one justify to a corporate overlord, to spend all the time and resources to refactor their code, upgrade libraries (assuming the libraries themselves upgrade), update build systems, perform extensive regression testing, and everything else, just for a version that would be EOL one year after what they just upgraded from?
The longer you delay upgrades, the bigger the delta, the more work it will be to update. Instead of doing two medium sized updates, one today and another one in 2030, you'll be doing a huge one in 2030. That might be a trade off you might be willing to make today, but you might regret it in 2029 when you're halfway through a multiyear migration project where all feature work is on hold.
The actual migration should you choose to do it was almost trivial. The official conversion script did most if not all the work. The changes were all cosmetic, like "print 42" changing to "print(42)".
I'd say the uproar was about the minorness itself. The syntax had these tiny but otherwise incompatible changes, and for many people, for no good reason. Fo example it was deemed that, as in the example above, print shall not be a statement anymore, but a function instead. Why? Unclear. Meanwhile, Python 3 offered few concrete improvements to many users
So I think the 2->3 moan, of which I was a party, was about that: minor, unnecessary changes that introduced little to no benefit, and yet backwards-incompatible.
If that were it, it'd have been over a lot sooner. For some packages, bytes vs Unicode strings was bad. For many more, layers of dependencies were an impediment. The trivial stuff wasn't the problem.
But for example a big stumbling block was the scientific programming community, from which I also stem. There were no major syntactical reasons there and yet people were up in arms, for the reasons I describe above.
Most people I know migrated only when python2 literally wouldnt work for them anymore, like they needed a feature of a library only released for python3.
print("goodbye, sweet prince")
Lots of abuse of how weakly/poorly/forgivingly typed Py2 is.
Doing the same processes in Py3 is actually headache inducing because of the much stronger typing system.
The LTS project[0] plans for a minimum of 5 year support from the date of initial release, so Aug 2026 is based on the Aug 2021 release date + 5 years. The rough-schedule on the LTS wiki also uses "June 2026" for now.
That's what we've been showing so far on https://endoflife.date/debian (With a disclaimer about what LTS means). Suggestions welcome!
The effort for implementation and package maintainers to support both versions was considerable, resulting in some notable packages only supporting one or the other.
In the words of the Python maintainers:
We needed to sunset Python 2 so we can help Python users by improving Python faster.
We released Python 2.0 in 2000. We realized a few years later that we needed to make big changes to improve Python. So in 2006, we started Python 3.0. Many people did not upgrade, and we did not want to hurt them. So, for many years, we have kept improving and publishing both Python 2 and Python 3.
But this makes it hard to improve Python. There are improvements Python 2 can't handle. And we have less time to work on making Python 3 better and faster.
And if many people keep using Python 2, then that makes it hard for the volunteers who use Python to make software. They can't use the good new things in Python 3 to improve the tools they make.
Of course, "We want this code to run forever the way it is" is a value option that a company can choose - just not with any language it wants. And somehow, people seem a bit reluctant to switch from Python to C to be able to keep running the same programs forever, guaranteed.
Effectively, however, python2 and python3 are different languages. Since they're interpreted, there's now a problem with '#!/usr/bin/env python' in your script: which python version is being invoked? Which language version are modules targeting? It's a mess.
But the poster child for screwing up a language version upgrade is still Perl.
The first 3.x releases were in no way good enough to replace 2.x and should not be counted like this.
Great that Debian finally pulls the plug on it, that means that most if not all of the downstream distros (such as Ubuntu) will probably follow suit, that's a big fraction of the whole Linux eco-system.
pip
Did you know pip can execute arbitrary code when installing packages?
Some would consider it cheating but I haven't got enough hours in the day. If I were to write some ad copy for Anaconda it would be 'in several years of frequent use this particular user didn't have to resolve a single conflict, it all just worked'.
Which runs counter to just about all of my other experiences with Python and its package managers/distributions/toolchains, some of those stories would end with 'and as a result the whole server was hosed starting from what should have been a routine certificate update' and other such nice adventures.
If you're writing Python code, `poetry` has been the least bad package manager in my experience. It has a lock-file, so it should help avoid supply-side attacks (like the PyTorch one from a few days ago [1]), and it means your local environment and CI/CD environment should have the same packages installed.
I'd also recommend setting the [`virtualenvs.in-project`][2] setting to `true`, to store the `.venv` in the same folder as your code, so that when you delete your code, all of the downloaded packages get deleted too. Otherwise they'll just stick around in your user directory and use up a lot of disk space.
[1]: https://pytorch.org/blog/compromised-nightly-dependency/
[2]: https://python-poetry.org/docs/configuration/#virtualenvsin-...
I know that Python 2 has stuck around for far longer than anyone wanted and that people were (are?) hesitant to migrate from 2 to 3, but I'm not sure what decisions contributed to that.
So IMO, basically what should have happened is accounting for valid use of features that 3 wanted to kill, and actually offering a path forward, from day one, so everyone could do a migration and actually have it work.
Edit: Also (but connected), 3.0 (and IIRC a couple thereafter) should have been marked as release candidates, because they were.
For us it was a change in string handling that stopped us being able to upgrade. Literally 10,000 things would have to be checked. We saw that red hat was supporting python 2 until 2025 so we decided to keep using python 2 until then. I think that once red hat stops supporting python 2 someone else will keep it running.
Python 2 still makes the world go round.
(I know someone who will pretend to know my situation better than I will advocate for some "solution;" no, it won't work in a way that won't break some of my priors)
I just wish they would stop iterating on the minor version so quickly. Why are we on python 3.11 (for workgroups, just kidding)? Is there that much of a difference compared to 3.7? 3.7 even has dicts with stable key insertion order and type hinting, so it seems pretty loaded if you ask me.
There's a lot of improvements being made to Python on these minor versions. So while the version bump looks minor, they can make quite a big impact.
3.11 improved performance by 25-60% (depending on use-case). 3.12 is set to improve performance yet again (Special thanks to Mark Shannon, GvR and Microsoft). On top of that there's a lot of quality-of-life improvements for the various parts of the std.lib.
I like this apporach very much, since upgrading to a new minor Python version is somewhat trivial
I'm in the minority here on HN, I can't stand the fact that something I use daily on Windows[1] and can continue to do so forever (because it's a compiled program, and the binaries will likely work for decades) can't run the same code in Linux, where the source is in python, and both it and wx_windows have made breaking changes.
For me, and my camp, it seems silly to simply write off all the code that conformed to the standards of the time, and doesn't properly bow to the changing fashions of the day.
It would be like declaring all HEX wrenches and tools to be obsolete one day, and insisting that everything was either 7 sided or heptalobe instead, for "reasons"
It seems completely irrational that Python 2 is excluded.
This isn't like a web site where there is an active party supporting the other end, this is code written in a language that had risen to popularity based on how well it was supported.
And that support has been arbitrarily withdrawn, and the HN crowd seems to cheer it on. I submit that this is part of why the year of the Linux Desktop just keeps receding into the future.
It is not about fashion, it is about resources and manpower. Supporting Python2 is not free, especially post-EOL, and a huge amount of community effort has gone into supporting it. It is not fair to demand that Debian ensure that vendor-unsupported legacy software continue working indefinitely, especially given the environment it would be running on is a moving target that will only diverge further from what was contemporary with Python2 with each release.
>This isn't like a web site where there is an active party supporting the other end
This is exactly like that. The "active party" in this case just happens to be the Debian community.
If you wish to continue using Python2, either stick to running it on an outdated environment such as Debian Buster that supports it natively (and accept the security consequences of doing so), take on the burden of porting it to modern Debian yourself, or seek out one of the commercial support options that exist for post-EOL Python2. You are not entitled to indefinite free labour.
I don't understand why the Python community doesn't just run python2 syntax with a flag, similar to the std=c89 flag in gcc.
Debian does not plan to maintain support for unversioned python commmands [1]. As much Python2 software assumes this is the case, continuing to provide these packages would become an ongoing burden.
Python2 has not had a security update since the EOL version 2.7.18. I am not aware of any current CVE's against this version but if one were found Debian would either have to withdraw the package or patch it for any stable versions of Debian. This includes maintaining the build infrastructure for Python2 to be able to patch it if required, bug tracker, etc. as long as Python2 is offered.
> What's the actual cost of not taking Python out of the repositories? I can still run Windows 32 bit applications in 2023.
I do not have numbers for this, but as the commercial Python2 support I was looking at (Activestate) is "inquire for pricing" I would imagine the equivalent community manpower/resource cost is significant. Just hosting the .deb in the APT repos is cheap - everything else around it is the expensive part.
So it is not so much "the cost of removing it" and more "the cost of keeping it around".
[1] https://www.debian.org/doc/packaging-manuals/python-policy/i...
The idea of having to test a language seems weird, either the interpreter runs, or it doesn't. Why should the Linux Distribution have their fingers in it?
Also, why does an interpreter need security updates? That's the job of the operating system, not applications, languages, or compilers. You should be able to run any random program in a safe and secure manner, no matter how evil it is.
It's the lack of capability based security that is the real problem with Linux.
Apparently there will be no Python 4 [1].
>The idea of having to test a language seems weird, either the interpreter runs, or it doesn't. Why should the Linux Distribution have their fingers in it?
Sorry if this was not clear - not testing the language (although you'd need to make sure the standard library works properly for anything with C bindings), but that the interpreter runs properly under each new version of the Debian operating system.
Debian would have to build and package the Python2 binary and related stuff from upstream. There are no Python2 vendor-supplied Linux binaries or deb packages. Officially continuing support would also include having to provide patches in case of security issues etc., fixing any breakage (I'm not sure what Python depends on but I have had friends suffer breakage between versions of Debian with newer versions of glibc and so on), etc.
>Also, why does an interpreter need security updates?
Things like this issue in Python's sockets library[2]:
"Buffer overflow in the socket.recvfrom_into function in Modules/socketmodule.c in Python 2.5 before 2.7.7, 3.x before 3.3.4, and 3.4.x before 3.4rc1 allows remote attackers to execute arbitrary code via a crafted string. "
There have been a number of vulns for Python itself over the years [3]. Not aware of any for the last version of Python2 but that may be simply for lack of attention given it's EOL and in theory everyone should have migrated to Python3 by now.
>You should be able to run any random program in a safe and secure manner, no matter how evil it is.
>It's the lack of capability based security that is the real problem with Linux.
Perhaps, and things like SELinux help, but it does not change the reality that someone has to maintain the software when these issues do show up.
[1] https://www.techrepublic.com/article/programming-languages-w...
[2] https://www.cvedetails.com/cve/CVE-2014-1912/
[3] https://www.cvedetails.com/version-search.php?vendor=Python&...
The problem is that SELinux is not an implementation of capabilities based security, despite appearances.
And of course cue the endless lines of fanboys screaming get used to the new reality, rewrite everything to Python 4 cus Python 3 is EOL and gone folks!
It's a huge usability fail on the part of Python that it's not obvious and easily memorable how to do this.
If you think about it, that's the only way that makes sense; if you wanted to go from, say, UTF8 to UTF16, you'd need to have an intermediate step that was abstract and formatless. Similarly, when you're operating on text, you don't want it to have a particular format; the format should only exist at the edges, where you take text in and output it. The format is a way to communicate with other programs, not components within your program.
My advice would be, if that doesn't stick in your head (totally legitimate), write down a code example and stick it in a personal wiki or something. I find it really helps me to understand something once and more or less copy paste it forever.
This is a key reason why people hated Python 3 so much IMO.
For context I’m a huge advocate of Python 3, except for this (and lambda functions, but that’s another whine session).
https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
A simplistic summary is that you always decode to unicode and always encode to bytes.
Another way to think about it is that Unicode is an idealized character model that is reified into a byte encoding, usually UTF-8, for practical usage.
It seems to me that this is something a lot of developers don't think too much about until they run into some obscure encoding/decoding issue, but it's kind of a fundamental aspect of programming that's worth spending some time grokking.
The problem is that encode and decode are directional constructs that change depending on context.
That’s a poor UI in Python.
It's tricky to look up a different solution you aren't familiar since the terminology is different. What I'm seeing currently doesn't really look that different to Python, but with different names for the types and such.
Taking a stab at it; This looks simple, and runes are great (similar to Python):
https://golangdocs.com/strings-in-golang
But something that bothers me is that converting an array of bytes to a string looks to implicitly be ASCII, or possibly UTF8. This document doesn't say.
This document acknowledges encodings and clarifies that literals are UTF8, but makes no mention of byte arrays:
I've poked through as many SO questions as I care to and am not clear on how to use different encodings. Looks to me like this is a difference in philosophy; Python would say, explicit is better then implicit, and I'm inclined to agree.
If I'm not understanding properly I'm happy to be corrected.
byte[] unicodeBytes = Encoding.UTF8.GetBytes(inputString);
// Perform the conversion from one encoding to the other.
byte[] asciiBytes = Encoding.Convert(Encoding.UTF8, Encoding.ASCII, unicodeBytes);
string outputString = Encoding.ASCII.GetString(asciiBytes);
utf8_bytes: bytes = bytes(input_string, encoding="utf-8")
# Perform the conversion from one encoding to the other.
unicode_string: str = str(utf8_bytes, encoding="utf-8")
ascii_bytes: bytes = bytes(unicode_string, encoding="ascii")
# The conversion could also be written as:
ascii_bytes: bytes = utf8_bytes.decode("utf-8").encode("ascii")
output_string: str = str(ascii_bytes, encoding="ascii")
Perhaps a convenience method would be nice, something like this, but it somewhat obscures the intermediate decode-to-unicode step:
ascii_bytes: bytes = utf8_bytes.convert(to_encoding="ascii")Isn't it true that encode always means text -> bytes and decode always means bytes -> text?
It's really sad that the Python folks committed suicide this way, all over a unicode fetish.
"Everyone has has a chance to migrate" . Doesn't work that way, just yesterday I spent a few hours trying to get scripts that are really good and would take me forever to migrate that are in python2 to work on an ubuntu based distro and this is with pyenv, custom venv,etc... just when you get python syntax worked out you are faced with deprecated or uselessly renamed imports and modules.
This is the shitty aspect of using foss stuff. I don't pay for it so I have to accept their decission. But in the broader sense, inability to influence things like this even with a ton of other people willing to pay for it is frustrating.
You know what would be great? If the community can vote with money pledges. So it you have 1000 people willing to donate $10/month to support python on sysv init then if volunteers don't want the money we get an external person getting paid 10k/month to support these packages even if the corporate types, developers and "devops"/sysadmin people have no use for it.
The free part of foss is expensive but I think the middle ground is to collectively pay for the support. I try very hard to avoid python2, I am not a developer and I very easily lose at least 1h/month on average to resolve these issues.
Do you all know why 2023 will not be the year of the Linux desktop? Because desktops are a commercial market and consumers/users cannot sway what the Linux desktop experience is. They either accept things as they are or pay for them by spending time reporting bugs, troubleshooting stuff, asking questions on random sites,etc... a lot of this has to do with change and maintenance requiring human time and resources. It's great when there are no issues and volunteers donate their time but it would be great to be able to pay with money also to support features or resolve issues. You can even make sure multiple people pay an equal amount to avoid wealthy people having too much or an influence but it still will not affect regular volunteer work.
I was hoping Ubuntu and Fedora would have such features.
And... you believe that Microsoft and Apple are swayed by users? More than Red Hat and Canonical? I don't buy that for one second.
That's true. Now look at 8, 10, and 11 and tell me with a straight face that they still care.
> Do you know why MS is bloated and "slower"? Because of all the technology they support from decades ago and all the interpperability layers in between.
Then surely the comparison is RHEL, where companies pay for compatibility and stability, not "we <3 bleeding edge" Fedora or "volunteers maintaining a coherent system" Debian.
Do you see the Metro start screen anywhere on Windows 10 or 11?
If they weren't swayed by user input, why did they throw away all that time, effort, and energy put into the the Metro UI and go back to the start menu paradigm?
> Then surely the comparison is RHEL, where companies pay for compatibility and stability, not "we <3 bleeding edge" Fedora or "volunteers maintaining a coherent system" Debian.
Unfortunately, RHEL and SLED are both targeting enterprise users, very little of that is desktop as well.
You know who else succeeded in the Linux desktop that isn't really considered proper Linux? ChromeOS on the ChromeBook.
If only these new laptop makers like purism or system76 took a similar radical approach where they would indeed listen to paying users more directly better than existing bigcorps, that would give Linux on the desktop a chance at market share gains that are significant. Then I would be comfortable gifting a Linux laptop to a non-technical user who can call a support line when having issues or needing a feature.
Stand-alone applications should use pyenv/venv because you can’t guarantee which version of Python is the system-wide one (and you don’t necessarily want to tie your app to the distro’s release schedule). Dependencies should always be in a venv as to not interfere with the system-wide ones or other applications.
If the removal of Python from distro repos breaks your things then it’s a wake up call and you’ve been doing it wrong the whole time. Fortunately it’s an easy fix.
That does not sound like a user friendly programming language.
The version compatibility of Python is a joke and has been after 2.7 to 3 to. Which is the main reason I don't use Python for anything but bash-like scripts.
Other languages or OSes that don't have their interpreter bundled forced you to think about the problem from the start and build a way to ship a stable version of the runtime with your application.
Python, as a server-side programming language is not too bad. You can standardise on a version and bundle that interpreter version with your application itself in whatever artefact you produce (whether a Docker container, tarball with Python binaries + source or entire VM image with the right Python version installed).
Desktop distribution is a pain, but that's not limited to Python; any language that can't compile to a native binary will have this problem - even Java, despite having much better backwards compatibility, still isn't prefect and sometimes requires an older JRE to be installed on your target machine.
If you want to complain about Python 2 being unsupported by the Python foundation, you're free to. But no distro can in good conscience keep distributing a thoroughly unsupported piece of software that's probably full of security bugs.
Regarding the "year of the Linux desktop" thing: Apple is much more aggressive than any Linux distro when it comes to dropping deprecated stuff and forcing their users to move on. Apple hasn't shipped Python 2 for a long time, they ship Python 3.
What lesson was learned? It seems like a weird corollary of the Streisand Effect: if you announce something's EOL, you'll get more clingy users that keep the product out of the grave, creating a drag on maintenance budgets, but, maybe, if you just stop maintaining and walk away, dependent users will either notice and switch away, or die themselves, and the death will ultimately be quicker and quieter.
(1) As an example: https://news.ycombinator.com/item?id=26168596
LOL that's an ancient bit of copy pasta. Here's a page from over two decades ago that was already describing it as a "classic troll".
Seriously though, that was one interesting language, the number of weird features (the editor is a string) that you'll come across in MUMPS make it fascinating just to study and wonder what inputs into the decision making process around the design trade offs were.
In case you're not trolling: FreeBSD optimizations used by Netflix to serve video at 800Gb/s [pdf][1]
Usenet activity is a very odd measure of deadness to use today.
Although maybe that is exactly what you mean: that the BSDs were already dying in the Usenet era and are still not dead? I think even if they were dying back then, over the course of the past 20 years they may have found their niches to heal and survive in.
Also poeple dont like to change shit that runs well, thats why we have so much legacy stuff around.
In the real world, it doesnt matter if your whole damn business is still running their warehouse system on a DOS program running in a god damn DOS emulator on a Win xp VM running on some linux box. What matters is that it works.
Those are from the 1980s.