GIL Become Optional in Python 3.13
geekpython.in
geekpython.in
I get that for people for whom Python is their main language, all these changes make their lives better. But for me it kinda just gets in the way. Personally, I wish Python would continue to be maintained as is, but stop changing so much. Stop breaking my builds.
What prevents you from using an old Python version?
With no python version manager (I don’t even know any off the top of my head).
Also virtualenvs only get you so far, I’ve come back to many projects on my machine with broken virtualenvs, I guess due to some dynamically linked dependency of python that my system no longer has.
:(
Try a tool called `pyenv`. It's pretty great. I pair it with the package manager called `poetry`.
Edit: Somebody's going to respond with the XKCD aren't they? I know about the XKCD and I nevertheless have high hopes for `uv`.
https://github.com/asdf-vm/asdf
And here's a list of everything it supports: https://github.com/asdf-vm/asdf-plugins/tree/master/plugins
Beside, "figure out which ones work" is a solved issue with a package manager such as poetry.
I don't especially like language-specific package downloaders either but that's not a Python-only problem, and there are tradeoffs to pushing the work downstream into more stable systems.
My current solution is a docker container which includes python 2.7 and a downloaded copy of every package I use.
I think the fact that it's not on the distributions is a major factor here.
Python changes because of three reasons: Addressing bugs and insecure patterns; support for new usage patterns in the community; updates for compatibility with newer devices and systems.
The first one is a given. The second and third are basically "the world is evolving, with or without you". Even if you built a VM on your perfect machine, with your perfect python and perfect libraries and froze everything for 10 years, the moment you connected it to the internet you'd be faced with "Ah, my perfect version does not support this new thingamajig being used on the web".
Freezing everything in time is just archiving.
None of this explains why python has the need to drop modules for the standard library.
They claim "nobody is maintaining them"… but they are well funded and can easily afford someone to maintain them.
I think at the core they don't care about backwards compatibility, but instead of doing it all at once like with python2/3 are now doing it little by little, creating a python4 without calling it so.
That's the unfortunate modern trend. The web folks that move fast and break things love living on the bleeding edge, and create strong pressure for everyone to keep up, and that the answer to library compatibility is "wait until someone complains, fix the direct reason, cut a new release, and let the dependents fix their own code".
You still have to find someone who both wants and can maintain them. That's not always a given.
And also, why? If I donate to the PSF, do I really want my money to go into the maintenance of ... checks notes ... MacOS 9 path manipulation functions (https://docs.python.org/it/3.7/library/macpath.html)?
Just because you're well-funded doesn't mean you get to spend money on useless things. And most of the python community would consider this useless. Nobody even made a pypi copy of it.
Python is a couple abstraction layers above that, it's not supposed to even be directly exposed to things that vary frequently between devices. That's a job of the OS, the C compiler, and the libraries (most of which just link to C ones anyway).
I don't think Python is perfect, I don't mind changes. But I do mind backward incompatible changes a lot.
Mature languages don’t tend to break working code. That was true when the quote was pronounced and that’s still true today.
Python is fairly unique in its propensity to be an unstable target. I personally can only think of the JS ecosystem has being worse but even then that’s mostly a library thing as the VM will run even the oldest JS code you can find. I wish the same could be said of Python.
From Python being brittle?
It’s one of the rare language which deprecates and removes API between minor versions. They have to provide migration guides with every release.
I'm also the type of person that doesn't do version-pinning as I think the concept is an anti-pattern that came from the JS world as a way to manage the chaos borne-about by the constant API changes, thousands of tiny libraries, and library dependency updates that it circularly-co-caused.
For such a long lived language to have this much immaturity is worrying. I guess the side affect of such popularity later in life.
I'm one of the people who was unkeen to move to Py3. My reason was mostly the numerous and totally unnecessary minor syntax changes (print -> function, map and filter becoming lazy, str/unicode etc) with no major benefits in the upgrade for my use case (data science), other than compatibility. Some of the new syntax was neither bad or good (what's so wrong with print being a statement? Add a printfn function if you really want one), some were IMHO outright bad (map and filter becoming lazy, reduce being kicked out to a module), str->unicode was maybe the only good one, but even that is controversial. The deliberate neutering of `bytes` was unnecessary too. Python 3 brought various implementation improvements and nice new syntax, which could have been implemented in a Python 2 syntax too (dict and set comprehensions for example)
So I saw no reason why Python 3 couldn't in fact be syntactically compatible with 2, and saw the whole thing as a bit of an upgrade circus: wanna keep using the language? Make these arbitrary changes because someone thought print ought to be a function.
You could argue it was the same disregard for stability as the now many versions of Python 3.
From the PEP[0]:
print is the only application-level functionality that has a statement dedicated to it. Within Python’s world, syntax is generally used as a last resort, when something can’t be done without help from the compiler. Print doesn’t qualify for such an exception.
At some point in application development one quite often feels the need to replace print output by something more sophisticated, like logging calls or calls into some other I/O library. With a print() function, this is a straightforward string replacement, today it is a mess adding all those parentheses and possibly converting >>stream style syntax.
Having special syntax for print puts up a much larger barrier for evolution, e.g. a hypothetical new printf() function is not too far fetched when it will coexist with a print() function.
There’s no easy way to convert print statements into another call if one needs a different separator, not spaces, or none at all. Also, there’s no easy way at all to conveniently print objects with some other separator than a space.
If print() is a function, it would be much easier to replace it within one module (just def print(*args):...) or even throughout a program (e.g. by putting a different function in __builtin__.print). As it is, one can do this by writing a class with a write() method and assigning that to sys.stdout – that’s not bad, but definitely a much larger conceptual leap, and it works at a different level than print.
But breaking one of the most commonly used statements for what is essentially aesthetics? No thanks. Introduce a function that does the same if you like.
That's precisely my point, this syntax change (and many others) was entirely unnecessary, put people off upgrading, then got them told off for being dinosauric luddites.
Frankly, it's not even that consistent. Is `del dict[key]` necessary when you can call `dict.pop`?
Either way, these would be fine discussions to have, but not when you consider the people upgrading thei codebases.
Reason for Python 4 detected!
I would never use Python 2 now, but I do understand why some people would choose 2 back when 3 was new (and even slower than 2!).
Would you not use Pathlib?
Rust has a similar thing with OsStr. In my opinion a clear type based separation between different kinds of strings (Text, data, os path, ...) is just required to write robust softwsre
I use python3 now by default, as it has fstrings, and probably some iteration things that I now rely on, and is faster for a lot of things.
But for the longest time the only difference that I bumped into was that print isnt a keyword anymore, but assert still is.
But it's good that it's Unicode.
(I think it was 3.3 when they compromised, maybe 3.4?)
Python is my main language and backward compatibility is far more important to me than performance.
Almost everything I do spends most of its time waiting for IO, or in a C library, or another process, or performance is unimportant. That is why Python is my main language.
I think it is possible that Python's popularity with people doing numerical stuff might end up making it worse for the rest of us.
I've been using Python since long before it became the go-to choice for numerics inside and outside academia, so let me tell you: It was even worse back then. The transition from 2.7 to 3.0 was both regarding the fundamental idea and the implementation the worst thing I have ever seen happen in any major language. This could (maybe should) have been a language killer. They should have just branched it out as something completely new. Van Rossum is totally right in saying that he never wants a Python 4 for that reason, so it seems at least he learned a little bit since then.
As systems started shipping Python 3, the convention among most of them became to leave `python` as a Python 2 interpreter and then introduce `python3` as a Python 3 interpreter. As they then stopped shipping Python 2, they generally just removed /usr/bin/python, so the only Python on the system was /usr/bin/python3.
In my own scripts to build Google stuff, I always had to make a folder with a single script called `python` which would contain just `exec python3 "$@"`, and then prepend that directory to the `PATH` when invoking Google's tools.
Google eventually did update their stuff to work on systems where the only Python is `python3`, but man this transition was rough. As I understand it, Google had good reason to do it their way at the time too; there was simply no consistency in how systems handled the 2 -> 3 switch. I suspect that this is due to a severe lack of guidance from the Python foundation.
Imagine if GNU released a new, highly backwards incompatible version of bash called Bash 6, and then some platforms kept /bin/bash as Bash 5 and introduced /bin/bash6, some systems replaced /bin/bash with Bash 6 and introduced /bin/bash5, and some systems removed /bin/bash entirely and /bin/bash6 was the only Bash on the system.
Python had the opportunity to be, and was, to some extent, just a standard part of a UNIX-like system, like bash. But now it's just another language where you need to care a lot about interpreter versions and how different distros package it differently and keeping code up to date to work with the latest Python and keeping Python up to date to work with the latest code.
This doesn't mean that Python is bad, but it would have been much more useful to me, and quite a few others, if it had stated a ubiquitous tool which is more like Bash and Perl.
A big attraction of Python is that it can do everything. Very useful if you need more than one of the above in a single project.
On the other hand I do feel that Python has been bad for my personal development because I can always find a Python library for any and everything it has reduced my motivation to learn other languages properly (i.e. actually using them for real work - I have learned a few in theory). I have been stuck with the same main (to the point of dominance!) language for far too long.
Yes, 2 to 3 was pretty painful, although it was not too bad for me because I was mostly able to delay moving until i was sure all the libraries I used had made the transition.
Just because the BFDL doesn't increment the major version doesn't change the fact that myriad breaking changes arrive in Python 3 PEPs.
This is purely an optics hack. Python 3.0 is very different from Python 3.13. Had the frog not been boiled slowly, it might as well have been Python 4.
The people who could stop this churn will not. It's a full-employment ticket for them, chasing dot-release upgrades and educating us about changes that we couldn't live without but somehow did for decades.
It takes a big person to say when something is good enough despite its warts. Contrast the Python churn with Linux and git: never break user space.
Windows 11 is a thing.
I didn't say things can't change. I said don't play games with major version numbers to window dress optics.
Survivorship bias. Python just barely survived the transition, and you only still use it now because it did make it. Take a look at Perl 6 to see a major language that failed.
Dealing with Python after mostly taking a pause around time 3.0 was introduced, I was shocked at the mess. And unlike with Perl 6, which took quite a clean break even before it decided to rebrand, you ended up with situation where both were around semi-equally, and there were "fun" bugs introduced in Python 3 that caused explosive bugs that we couldn't figure out in beginning because things would always work on developer machines...
Does it by any chance a process started by multiprocessing? /s.
On a serious note, I feel like this is a valid usage and as someone who really like the no-gil approach I still would want python to handle the gil and no-gil flags in a better way. I don't know yet how this will manifest in real life and if keeping gil will be an option on the long run but it is a trade-off. There is no fundamental reason why python cannot have proper multithreading support.
It drives me _nuts_ when I find a simple library to do a small-ish thing that I don't want to write/maintain myself only to discover that it uses numpy or pandas because they're "fast".
I wrote some python programs that did multithreading, but they didn't work and I had to go elsewhere.
So... it is possible you won't find many people in your sample that disagree because the folks that went elsewhere... aren't part of the sample.
You don’t have to install the latest version if you don’t want to. If you use a version manager, you can keep using whatever version you want for many years.
i don't think it's too surprising that versions differ. there are good tools to manage the python version used -- i personally like pyenv.
As I said, a lot of the value I see in python is in its ubiquity. This disappears if you can't use the system python.
But if you value that ubiquity more than your sanity, I hope you find and fix any breakage yourself and not pester other devs for support.
+1
Most Python use case today is some what on the similar lines as Perl. Having a stable large install base. Availability of libraries, backwards compatibility, performance matter way more than more features at this point.
There is no new replacement language in sight, so Im guessing we will be using Python for long into the coming future.
I also hope Python had something on the lines similar to CPAN. We are decades into the journey, and Perl still shines like the Sun in this regard.
Part of me feels sad Perl 5 didn't go the way Python 3 did.
Especially considering how a major distro[1] that defaulted to Python 2.7 only just was dropped by many[2] - but not all, because some "heroes" jumped at the last moment to provide binary compatible support options
[1] CentOS 7
[2] Especially if you wanted to be in any case acceptable to sell to FedRAMP-requiring or similar clientele, but in Europe NIS 1 (already in force) and NIS2 (starts enforcement this year) ban software that is post end-of-support
What do you mean? I've been using python since 1.5 and it is more ubiquitous today than it has ever been at any other time in the past.
I remember Python around 2.2 becoming ubiquitous and "obvious" option as "better than perl and also certainly available". Python people would even trash talk other communities for not being able to "just download the libraries from distro repo" or needing specific versions whereas you at worst grabbed updated python for your distro.
(The following is a somewhat chronological rant)
Fast forward few years and me getting out of university which for various reasons was (apparently blissfully) Python free. In between we get through the "huge"transition from Ruby 1.8 to 1.9/2.0 (had to help professor update coursework even).
The 2->3 transition interrupted that, and I am suddenly facing that a) python3 actually got released b) despite several years, it's not even guaranteed default c) you now have to explicitly use python3 or python3 in shebangs because you don't know what version of python is default d) virtualenv wtf e) there's still new code being written depending on py2 f) the library situation can be a maze.
Since I mainly worked as "DevOps" person, not a python programmer, I get to deal with increasingly borked python deployments. Suddenly authors names in comments cause deployments to fail on servers but work on dev machines as well as mine. Took long to figure, the same code working when connecting over SSH vs not working when started by systemd shows it's python deciding encoding by locale and barfing on 8bit characters (including utf-8).
Virtualenv, increasingly necessary multiple python versions, all driving increasing replacement of python in admin/management tooling. Chef and Puppet (finally) laughing from their omnibus setups at ansible exploding depending on what python is default on target distro and people who pushed ansible confused because "it only uses SSH, there's no agent". Even more movement to Golang because of static binaries.
Virtualenv becomes the effective default, forget about installing packages through distro (subjective take, I know, this is a vibe I know it's still possible).
Python 2 goes EOL and out of support but it's still default on distros used by paying customers (this changes - but not fully - this year on June 30th... Thanks to government regulation - thanks Obama /s)
I now mostly see python deployed with not just batteries but the whole jungle included (Docker). If it's s good container it doesn't have a complete copy of development environment.
Increasing mess with packaging. Distros reducing python dependencies, system python increasingly seems mostly for use only by OS components. Python equivalents of rbenv/rvm increasingly suggested as "right tool". Package build items exploding because transitive rust deps.
Customers sometimes demanding OS installs without Python. Increasingly encountering distros with no python at all.
Python packaging still feels worse today than rubygems were a decade ago.
Finally increasingly suggesting to prospective projects that benefit/cost of python goes below 1.
I used to love python back in 2002-2006 timeframe. Rails making it so damn easy to handle some projects and finally starting to grow Lisp (thanks to a detour through Haskell due to XMonad) made me look away. Even "ML ecosystem" mostly gives me more reasons to rant against Python (Tensorflow packaging as PIP wheel makes me problems even when I am going to use Python.
So, yeah, lots of code in Python, ubiquitous python jobs. Retreat from "write your tool in python, it will be easy to distribute, every distro effectively preinstalls python unless it's some LFS or Slackware weirdo"
There was a consensus after the Python 3 debacle of “No major breaking changes”. We seem to have lost that because of moneyed interests and that’s sad.
Who does no-GIL benefit? For the majority who use Python for single threaded code, no-GIL will make their code slower because a thread-safe interpreter is always going to be slower than one that can assume ST operation.
For the minority who want parallelism, there’s two other options: OS processes and subinterpreters. If you can use either of these then you will get better performance with a GIL for the same reason.
So no-GIL will only be faster for a minority of a minority who want parallelism but can’t use OS processes or subinterpreters.
Meanwhile everybody else writing libraries has to make sure that their code is no-GIL safe, to support this tiny minority, and if no-GIL ever becomes default then everybody else has to do something to turn it off.
It’s such a stupid idea.
Yes, that was my point.
> For the majority who use Python for single threaded code, no-GIL will make their code slower because a thread-safe interpreter is always going to be slower than one that can assume ST operation.
I'm almost sure the python developers said that they will compensate the slow down with other optimizations, so that you'd never have single-threaded performance degradation version-to-version.
> So no-GIL will only be faster for a minority of a minority who want parallelism but can’t use OS processes or subinterpreters.
One would hope to 1. open new use cases for python thus attracting developers that would have otherwise not given the language consideration and 2. other users could benefit from new optimizations that could be implemented further down the dependency stack.
Of course there's no guarantee that that will materialize, but the idea the adding support to an established, lightweight and well-supported concurrency primitive is so obviously a "stupid idea" shows to me that your (rudely expressed) opinion is entirely self-centered and nearsighted.
I might add that the move from python 2 to 3 was incredibly painful, but I assume most agree (with the benefit of hindsight) that it was entirely correct.
Those optimisations are not there to compensate anything; they will improve performance of single-threaded code with or without GIL.
I think running inference engines is one particular case of that. You can train and tweak your model using python with its wonderful pytorch and numpy. Then at some point when scale and performance matter then this becomes a potential problem.
Companies like Facebook also have huge interests in speeding up python as they have a lot of stacks written in python.
To also enable this as default in the coming years is crazy.
I would love to hear what Guido thinks about this.
Also there's no way to get access to pypi and fix the abandoned libraries. So they get patched in distributions and remain broken forever on pypi.
Python almost never does breaking changes in non-major versions, and only when necessary.
And even at that, "Python" didn't break, only those (abandoned) packages that depend on distutils, and they already got/are getting fixed/removed. Honestly this isn't "breaks at every Python update".
All those of us who aren't using those distributions don't even see the abandoned libraries in the first place. (I asked you guys for a list of the libraries.)
Another example of breaking changes in the Python ecosystem: when Numpy updated to 2.0 (with breaking changes) we had a package that depended on `numpy >= 1.something` but it didn't actually work with Python 2.0.
You can say "well they should have known to use the very unobvious non-default syntax 'numpy >= 1.0, < 2'" but even that isn't right because it turns out Python packages don't use semver; they just deprecate things and then remove them at arbitrary versions, so there's no reasonable upper bound you can actually put there.
That was certainly my experience working on TruffleRuby, and I doubt Python is very different. We found a few bugs because we didn’t have a global lock, but they were often problems that showed up in a soak test on an implementation with a GIL, just not quite as often so they had escaped notice and were likely causing real failures in production somewhere.
Basically it won't become a default until after it works. Which is the opposite of crazy; it's very reasonable.
Guido has spoken out on this and I don't think he's against this. It's more that he's in favor of stability and moving forward.
The challenge with python is that you can't do a lot of things in it very efficiently that people keep on doing in it anyway. Like trying to use multiple threads. Making those things work a bit better is not a bad thing.
There's a simple solution for code that depends on the GIL, which is simply to don't do what's fairly pointless with the GIL anyway: using more than 1 thread. The GIL only serves a purpose if you use more than 1 thread. And since it makes doing that kind of impractical anyway, most python code doesn't need the GIL because it is single threaded. That code will only break if the GIL is gone and multiple threads are used. Simple solution if that affects you: don't do that.
Everytime I've needed just a bit of multicore performance and have gotten stuck using multiprocessing sucks - I'd much prefer just to use threading and not worry about the pickling process.
https://www.infoworld.com/article/2338862/python-moves-to-re...
If your pure Python code fails without the GIL it can probably fail with the GIL, and testing it without the GIL might help you find those bugs a lot quicker.
No, if you remove the GIL there is a whole class of bugs that will appear in code that was safe with the GIL in place.
So if you're expecting something better than that, you will be disappointed.
[1] https://developer.vonage.com/en/blog/removing-pythons-gil-it...
I don't know what the plan is, but they haven't succeeded (outside of the corporate world, really, for C# and the functional world in F#) in making modern tooling and languages people want to use.
Yes, that's why having something like Pandas use it would be better than getting all users to write their own version.
I am pretty sure you are going to say there is a reason this cannot be done, would just like to know what it is!
There's packages to typecheck at runtime, even to check the parameters to a function.
As they are not enforced at runtime, you can easily return bollocks and not know.
I really like how c# does it, which is have strict typing on by default, but allowing you to turn it off for things where you're wanting to be loosey goosey. Not having to do a bunch of type checks on every operation would also speed up a bunch of things inside python
However, that would make it a different language. perhaps python 4? (ducks)
To those who want to turn Python into Golang/Rust/Haskell/Java ... guys, other programming languages are available.
Anything to avoid the over complexity associated with delivering content with nodejs it’s like you missed out on building your computational foundation.
It’s common that as project grows larger and more unwieldy, people ask for more warranty and help from their tooling. I think Python position on this strikes a good balance - provide the infrastructure for adding type hint but delegates type checking to a third party.
Lisp compiles to native code since 1962, having a JIT is pretty much given.
I only mentioned Common Lisp, because of the wide library, having already the implementation experience of Lisp Machines from Xerox PARC, TI, Genera, and being as dynamic as Python if not more so, already to sidestep the usual dynamism excuse for lack of Python JIT's.
Compiling to native code isn't JIT, it was very common to run just straight interpreted code on the Lisp Machines (since compiling took a long time, and then to load the final file object on slow disks).
Feel free to show us the manuals of those implementations without a chapter about (compile ....), (disassemble ....) or similar.
The Lisp Machine compiler is incremental, but it is not a dynamic compiler, it never changes the compiled object when it has been compiled during run-time. This is similar with all Lisp implementations, and even Python. Code is also not compiled by default on the Lisp Machine, it is interpreted.
https://docs.python.org/3/library/functions.html#compile https://docs.python.org/3/library/dis.html
Python has all the building blocks that Lisp does in this regard, and has had for many many years. But I'm not allowed to quote the Lisp Machine manual on the topic, since it has chapters on the compiler so I'll jump out of this conversation now...
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec...
https://web.archive.org/web/20201213195043/ftp://publication...
Dynamic compilation (i.e. compilation during execution and/or modifying the emitted compiled output during run-time -- depending on which school one prefers) was never a thing on the Lisp Machine, and is not even a thing in modern Lisp implementations like say SBCL which just does static compilation (compiled object is never modified when it has been compiled).
(print-herald)
LM-3 System, band 4 of AMS-LISPM-2. (LM-3)
2048K physical memory, 16127K virtual memory.
Experimental System 300.0
Experimental Local-File 54.0
Microcode 323
AMS Lisp Machine Two, with associated machine FS.
(defun foo () "bar")
FOO
(compiled-function-p #'foo)
NIL"The earliest published JIT compiler is generally attributed to work on LISP by John McCarthy in 1960"
As for the LM-3 example, I would rather be proven wrong by you pointing me to PyPy as counter example, but the poor fellow not even for this comes up.
LISP I had an incremental compiler. The user calls the compiler with a list of functions or function definitions to compile. It's not compiling dynamic by the system and also does not use information about the running function (call statistics, call argument statistics, ...).
There is no "Just in Time" functionality provided. The developer is responsible to decide what code to compile and when to compile the code. A function needs to be compiled by the developer before it is invoked, otherwise it won't run compiled. In a JIT compiled setting the system decides when to compile the code: either on start or during runtime triggered by the system. The code will be "Just in Time" compiled.
This Wikipedia article explains the difference:
https://en.wikipedia.org/wiki/Dynamic_compilation
"Just-in-time compilation is a form of dynamic compilation."
"Unlike dynamic compilation, as defined above, incremental compilation does not involve further optimisations after the program is first run."
Later Lisp systems may have aspects of JIT compilation. For example some CLOS implementations optimize code (-> method combinations) at runtime, possibly via compilation.
"The earliest published JIT compiler is generally attributed to work on LISP by John McCarthy in 1960"
I can naturally also go hunthing for those papers from my digital cellar.
https://github.com/google/pytype https://github.com/google/pytype https://github.com/microsoft/pyright
By that I mean if you do
foo = {"a": 2}
foo["b"] = 3
I guess you could argue most Python is not well written, but then most Python developers are too amateur to use Pyright anyway.