Python Language Summit: Python Without the GIL
pyfound.blogspot.com
pyfound.blogspot.com
One of the saddest things about popular open-source projects like Python is how inevitably the maintainers become worn down and jaded over time, due to constantly filtering and protecting Python from dumb or nonsensical ideas.
Then when something like this comes along, a true game changer, maintainers have no energy or goodwill left to collaborate on these infinitely important initiatives, like removing the dogdamn GIL. This is way more significant than everything I'm aware of that's happened since the birth of the language. All other things have been, trivial miuntiae in comparison.
It bears repeating: "everything else has been trivial minutiae in comparison to a GIL-ectamy for Python."
It cannot be understated, removing the GIL will be a HUGE deal for Python! It's an ugly wart which has existed for about 30 years, and nobody else has produced and delivered a working solution to the community.
I wish Team Python would welcome GIL Eradication with open arms and a supportive attitude. This would look like focusing on helping identify and implement solutions to the impediments rather than simply pointing out problems and then helicoptering away.
If it's such a huge game changer why don't some of the large enterprises which rely on Python fund this work?
edit: not sure why I got flagged, HN. Software engineer hours == funding.
Otherwise, there’s making a fork and attempting periodic updates from python. That’d be a huge undertaking though.
FTA: “”” A lot of the value of Python is the ecosystem, not just the language… CPython really leads the way in terms of the community moving as a block.
"Removing the GIL is a really transformative step. Most Python programs just don’t use threads at the moment if they want to run on multiple cores. If nogil is to be a success, the community as a whole has to buy into it."
– Sam Gross”””
[0]https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
[1] https://docs.python.org/3/library/multiprocessing.shared_mem...
I would hazard a guess to say that 99% of production Python is people doing the equivalent of welding, but there is also this 1% who want the GIL to be gone so they can repurpose Python as a tool for a whole new class of problem solving.
It will be an exciting future for them.
For us welders, we’ll carry on gluing stuff together blissfully unaware of the GIL. Async and non-blocking IO are great. I don’t think I’ve ever needed compute-concurrency, not in Python anyway.
As for the C, C++, Fortran libraries with Python bindings, any language with FFI can call into them as well.
I would say, Python's adoption while lacking performance is what is now building pressure to avoid specific Python communities to leave Python and migrate into one of those ecosystems in search of a better mix of productivity/performance, without being forced to use two languages.
And Clojure also has a .NET implementation.
But it is being continuously kept up to date by Cognitect, along with active development on a clojure-clr-next version. Maybe it has a bigger population of non public users.
This isn't a case of "x" or "y". There is literally nothing valuable about the GIL, it's an ugly hack of a vestigial appendage. Perhaps the reason I'm familiar with it is because the lack of elegant MP threading in Python perturbed me for years, until I was introduced to Golang.
Python devs generally don't want to use Java, JavaScript, etc. And the Go ecosystem is good but not as rich as Pythons.
Anyhow, take care pjmlp. Until our paths cross again I wish you all the best!
JavaScript, Java, .NET, C and C++ are where to look for, if you want to count package numbers, for AOT/JIT languages.
Some are so compute heavy that massive amounts of time is put into tuning them (sort algorithms, fast-fourier transforms, etc).
Most fall somewhere on the spectrum between those extremes. If you can speed up your program by a factor of 20 by adding threads, Python can cover a bit more of that in-between spectrum.
Just because python isn't the fastest language in the world, does not mean that making it easier / possible to write faster python is worthless.
I ported it to Python, with a totally naive simple single threaded loop. It worked perfectly. 25 updates a second, forever. No C code, except the interpreter doing its thing, and some GPIO code.
That's not slow as molasses.
> My page load time dropped from 77ms to 47ms when I ran some of the code in parallel.
I suspect some of us welders will also find some improvements here.
More than 5x can be achieved with the GIL included (and has been promised, by the new Microsoft Python-speed-up initiative and that Python dev who made a similar proposal).
If there wasn't the GIL, I could just create a thread pool and be done, and Cura could continue to be a delightful mess. :-)
No, I don't, and can't find any writing about this at all, do you have any links? Because that sounds amazing
This:
generate fullerene "arc weld"
I have. Some very inexperienced “senior engineers” at my last gig thought it would be fine to build an analytics platform in Python because “Pandas will make it fast”. Unfortunately even modest datasets would cause timeouts and even seize the event loop, while a naive Go prototype would finish in a few hundred milliseconds.
That is just not true.
1 - This helps parallelize regular python code too, like the kind that goes into writing web services.
2 - While you'd still write native extensions to get optimum performance for single threaded code, having the thread dispatch layer in python makes parallelizing execution convenient and robust. For example, see the comment from people who maintain scikit-learn below. I'd love to see python get composable task parallelism like Julia where you can freely spawn threads within threads without worrying about their lifetimes.
link: https://github.com/colesbury/nogil/issues/36#issuecomment-10...
> helps parallelize
> convenient and robust
So nice but not too noticeable?
What percentage of compute time is python?
What percent of Py compute time would be cut?
Who's using python?(banks, stem researchers, everyone else)
> ...bunch of questions...
Yeah, if you claim that something is a "game changer" you should provide some arguments, not questions/"help determine".
It might very well be a game changer but thud far you have only suggested there might be an argument for that, if ...
Edit: For backend services it would probably put Python around one order of magnitude away from Go in terms of memory usage to serve the same load, enough for teams not to consider switching.
Currently, that doesn't do much. Trying to do this with a processPool either works, or becomes a horrible exercise in fighting with pickle or whatever other serializer you pick.
What would I do differently? Actually make engineering choices based on the current project status and priorities without having those choices made for me.
https://discuss.python.org/t/official-list-of-core-developer...
https://pyfound.blogspot.com/2021/07/ukasz-langa-is-inaugura...
Because lots of code in python relies on the GIL to act as a synchronization primitive for threaded code (either explicitly or implicitly) and removing the GIL will break that code in subtle ways and there’s no appetite in those companies to fix that.
This is the first suggestion that doesn’t degrade single threaded performance.
If "Python 3.11+faster-cpython" didn't also remove the GIL, then it didn't negate anything he's trying to acomplish. He wasn't going for a faster Cpython alone, but for a Cpython without GIL.
Why is that unfortunate? If his patch being slower than python 3.11 isn't acceptably fast despite being faster than every other python version before that then python was never acceptably fast to begin with.
Linux got rid of the BKL ages ago, that the Python community still holds onto the GIL as if it was some multi threading holy grail isn't even remotely funny anymore.
But due to the leadership/community model/aversion to big changes (except for alienating people with 2 to 3 changes) none of this was in the context of CPython, they were independent versions, that never caught on, and didn't leave anything (or much) behind to the regular CPython when they folded.
[1] 3.7.0 is the first major release following the removal of the config option in https://github.com/python/cpython/issues/75551 .. there's also a 2021 comment about "This has unfortunately turned out to be a blocker on getting WASI support as there's not direct threading support in WebAssembly."
The 2021 workstation I am typing this on has 16 physical and 32 virtual cores, and I expect core counts to continue upward this decade. While the CPython core devs may be excellent programmers and do a good job at maintaining Python, they clearly have a bit of trouble with cost/benefit analysis if the complaint is that the test count will double in CI for a ~16X increase in the amount of code that can be run in parallel on even consumer machines. Yes, yes, I know that this does not mean my programs will run ~16X as fast. Yes, I know there are other objections. But this is an order of magnitude away from being an actual blocker and the fact it was brought up at all as an objection shows a fundamental disconnect between the small potential costs imposed on core devs, and the huge potential upsides for Python users everywhere.
You can already use multiple cores by writing C extensions that release the GIL and with multiprocessing. That double CI cost and additional work isn't just for core python but for the entire ecosystem.
Don’t use Python is also my preferred recommendation when I encounter Python.
1. Learning a new language is non-trivial for many people (and don't sneer - it's about time not competency)
2. The ecosystem matters. If the code you want to interface with is in Python then "don't use Python" is just glib.
I'm mainly proficient in Javascript, Python and C#. My choice of language is rarely based on "which is best for this task?" but mostly "want do I need to run on and interface with?"
True shared memory threads within a single process would be a major boon.
I chose Python to not have to write C...
The even more sucky part is to distribute them and make sure it works on every OS, without every user having to apt install build-utils before they can pip install your package and then spend rest of the day debugging some rare compilation error because of a header file mismatch with what’s installed on the system. The python packaging space is already complicated enough even without considering native modules.
b) Introducing native extensions means deployment and distribution becomes more difficult, and introduces a whole new and large class of issues and caveats into a project.
c) Native extensions are not written in Python. (Yes, Cython exists, no, it's usually not a good idea to write more than glue in it).
⇒ I don’t see this as a team of worn-down maintainers, but rather as a team of seasoned developers, who know you don’t turn a project with millions of users around on a dime.
One question I miss is whether the set of locks this change introduces is the optimal one.
I would think that, if it were to be improved upon in the future, C extensions would have to be changed again, so you’d rather not do that often.
Hasn’t this gil/nogil saga been going on for ~decades? What about fixing packaging or limiting the C-extension interface? I have enormous sympathy of what the Python maintainers are up against, but “on a dime” seems hard to justify.
That's numpy, pandas, sklearn, and friends.
[1]https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD...
> Why need concurrency in normal python scripts let’s days for web scraping or machine learning ?
I had a few scripts where I had to process over a hundred files, just running that in parallel can reduce a job that takes minutes to seconds.
From your link:
"Stripping out some of the GIL-removal related changes would result in even faster performance on the single-threaded pyperformance benchmark suite. [...] The resulting interpreter is about 9% faster than the no-GIL proof-of-concept (or ~19% faster than CPython 3.9.0a3). That 9% difference between the “nogil” interpreter and the stripped-down “nogil” interpreter can be thought of as the “cost” of the major GIL-removal changes."
So it seems removing the GIL has a negative impact on single-threaded code, with the version that has both the GIL and the unrelated optimizations being 9% faster.
That's not a noob question. That's a loaded question, from a perspective of knowing what people use Python for.
People use Python for all kinds of things, and a lot of them would be faster if they were able to take advantage of multithreading without the GIL.
Well, if they didn't, we'd still be stuck with Python 1 or 0.1.
Why is the GIL suddenly where Python should keep "being what it is", and not any of those tons of changes, from 0.1 to 3.10?
Especially since the removal of the GIL doesn't change any spirit/essence of Python - just makes it faster.
Python wasn't conceived as "having a GIL" being some essential part of it, it was just a bad tradeoff for implementation convenience made back in the day where common multi-core machines were 20 years in the future...
They are trying to strengthen corporate influence and discourage any power of traditional independent open source developers (who have contributed most of the useful features as opposed to corporate churn).
We do not know what is going on in the background, it is proxy wars all over the place. It is possible that they want to reduce Facebook influence, that their code bases do not need this feature, etc.
Just use C++.
Here's a 2007 essay by van Rossum on the topic: https://www.artima.com/weblogs/viewpost.jsp?thread=214235 .
> I'd welcome it if someone did another experiment along the lines of Greg's patch (which I haven't found online), and I'd welcome a set of patches into Py3k only if the performance for a single-threaded program (and for a multi-threaded but I/O-bound program) does not decrease.
Sometimes you just get lucky.
No, it’s not. My comment applies to single threaded performance as well (I wasn’t even thinking about the GIL when I made my comment)—why isn’t Python as fast as modern JS implementations, for example? The answer isn’t “JS has multi thread support”.
Moreover, a slim C extension interface is about flexibility so you don’t paint yourself into a corner when you don’t know what the world might look like tomorrow. Further, you can have a very rich C extension API without exposing the entire interpreter (e.g., h.py). Further still, Python has broken compatibility several times since its inception, so the idea that this was cemented in 91 is nonsense.
Because the project has explicitly targeted implementation simplicity, largely successfully, for almost 30 years. The internals are a joy to work on, and unsurprisingly the CPython Git repository has 4x as many contributors as v8, despite CPython contribution being largely voluntary and v8 contribution being largely commercial.
Even if performance were an explicit goal, it's important to remember v8 required absolutely massive funding and top tier engineering support to make it happen at all. The most comparable equivalent in the Python world, PyPy, was the product of an extremely dedicated group of mostly doctoral researchers working against incredible odds. V8 only has 2x as many contributors as PyPy. I hope by now you are recognizing a theme: the reason the language is so successful is also the reason we are here complaining about it.
There have been teams in at least Google and Dropbox who proposed major upheavals of the interpreter in the past. Both failed in large part due to the complexity of their proposals compared to the performance gains on offer
> The most comparable equivalent in the Python world, PyPy, was the product of an extremely dedicated group of mostly doctoral researchers working against incredible odds
"incredible odds" refers to compatibility with the CPython C-extension interface, which is exactly what I'm talking about.
> There have been teams in at least Google and Dropbox who proposed major upheavals of the interpreter in the past. Both failed in large part due to the complexity of their proposals compared to the performance gains on offer
No, they failed because they had to work within the considerable constraints imposed by historically bad decisions (such as the C-extension interface). The proposals need to be complex because they can't break compatibility.
> I hope by now you are recognizing a theme: the reason the language is so successful is also the reason we are here complaining about it.
Not at all! A narrower C-extension interface doesn't imply that C-extensions would be more difficult to write. There are no downsides to a narrower interface (apart from breaking compatibility, but we're positing a world in which this decision was made in 2008 or earlier).
The real theme here is that historical bad decisions + compatibility guarantees add significant complexity to every single improvement if they don't preclude them altogether.
I am not sure about it. Python explicitly trades performance for simplicity and the GIL simplifies a whole lot of things and because of it, there are multi threading issues that will never be encountered by Python code.
In addition, because of Python’s popularity, there is a massive amount of code written in Python (and other languages as extensions) and any change cannot introduce a bug.
Finally, there are other alternatives to Python that are more performance focused.
If you are having huge issues with Python performance, maybe you are using it in a matter it was not designed for.
It looks like they are supportive and keen on the changes? From the article:
> Gross's proposal was greeted with a mix of excitement and robust questioning from the assembled core developers.
It's definitely a huge opportunity, no doubt. But it affects the C ABI[1] for Python itself, something that everyone in the conversation is aware of, and that would have implications for all distributions of Python.
Add in the importance of effective code review for a 20k LoC changeset, and it seems there are good reasons to be cautious despite the optimism and excitement.
[1] - https://docs.python.org/3/c-api/stable.html#stable-applicati...
I would argue the opposite: it's the secret to Python's success. It might even be my top example of how "worse is better" plays out in real life.
I agree, the GIL feels like an ugly hack. The software engineer in me wants to hate it so much. And, now that I'm on the far side of a successful transition to a data science role, one might think that I hate it even more, yeah? Because the work I'm doing depends so very heavily on compute performance and parallelism.
But it turns out, nah, I'm coming to like it. It's an ugly hack, but it's the best kind of ugly hack: one that gets the job done.
Because I'm pretty sure that the GIL is the secret sauce that makes Python C extensions work so well. Without it, it would be much more difficult to write safe, correct C extensions. Doing it without introducing race conditions that destroy memory safety would be a black art. So people would probably do it less. And that probably means no robust Python scientific computing or data science ecosystem, because that stuff is all C extensions.
We could instead use a C FFI, like it's done in other languages. Java, for example. But Java having to use an FFI and Python being able to use its C extension mechanism is exactly why Python has eaten all of Java's Wheaties in the data space. The marshaling costs of talking back and forth across that boundary are just too high. Copying goes up, locality of reference goes down, cache misses go up. You saturate the memory bus more quickly. Once you've done that, it doesn't matter how many threads you have running on how many cores. The bottleneck is happening outside the CPU. Top will happily tell you those cores are working hard, but it turns out that what they're working so hard at is sitting around and waiting.
This isn't just theoretical. Last year I replaced a heavily parallelized compute-heavy batch process written in Java with a Python implementation that got the work done in less time despite being single-threaded. Sure, the Python implementation was basically a little scripting on top of a library written in C++, and the Java one was pure Java. But that's kind of the whole point. I also know that, back when I wrote the original Java implementation, I tried the same trick of farming the calculation out to a C++ library, and it actually made the throughput even worse. JNI is a harsh master.
And besides, as others have said, numpy & friends give me most the parallelism I actually need, for free.
Maybe it hurts other people more? Maybe Web developers? But there's a part of me that thinks, if you're trying to do that kind of work at scale, making Python go faster is perhaps barking up the wrong tree. There are plenty of other languages that are statically typed (reducing pointer chasing and branching can increase your throughput without giving Amazon more money in the process) and don't even need a global interpreter lock in the first place because they're not interpreted, either.
I could imagine getting pretty jaded as a python maintainer having to keep pointing out why the latest attempt won't work. I think the onus is on those who want this to demonstrate that it can be done successfully and lay out a plan as to how it can be put into the real world without causing chaos in a python ecosystem that has only relatively recently got over the trauma of python 3 (which is the bit I doubt is possible).
Edit: To be clear, I don't think anything that consists of a set of proposed changes to the cpython interpreter is even beginning to attempt to think about the implications for the ecosystem, which is where the actual challenges are, and I'm assuming that's what the "Just creating a PR" comment is saying.
It worked! I was stunned at how easy and effective it was. My page load time dropped from 77ms to 47ms when I ran some of the code in parallel.
I wrote some notes on that here: https://simonwillison.net/2022/May/6/weeknotes/#nogil
The socketserver.BaseServer class sets self.__shutdown_request in one thread and expects it to be picked up by another. In the Java memory model this variable would have to be marked as volatile (or the methods involved as synchronized) to make sure that the other thread will actually see the change, so unless Python implicitly make every variable volatile (does it?) this wouldn't be guaranteed to work (although it would probably mostly still work most of the time except for when it mysteriously doesn't).
The nogil development uses biased reference counting, with an atomic lock between threads, and doesn't change that behavior.
As I understand it, a different thread running on a different core could have a cached version of a value which wouldn't necessarily be updated unless some instruction is issued to synchronize with main memory, which in Java is done with volatile or synchronized.
Also, if some optimization is implemented that reorders instructions or eliminates the variable update entirely, that could also prevent the other thread from ever seeing the updated value. This is also solved by using volatile or sychronized in Java.
Is every variable implicitly volatile in nogil Python? Or only object attributes? Or have I completely misunderstood some important aspect?
Edit: I suppose modifying the reference count might cause implicit synchronization similar to the piggybacking technique [1] in Java, making this a non-issue?
That said, I am not a good source for truth on this. But it feels like so much code would break if this weren't true that I don't think it would have gotten to this stage.
FWIW, here's the primary description of how it works - https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD... .
Here's a patch the author himself wrote to fix a spot where this change break's numpy's thread safety: https://github.com/colesbury/numpy/commit/2ad41a1fb8b0c28fa8...
Maybe that's the only one? Maybe it isn't? But I think the point still stands that people saying this has the potential to break existing Python packages in subtle ways are not just being hyperbolic.
> To mitigate compatibility issues and improve debugging, the proof of concept can run with the GIL enabled or disabled controlled at runtime by an environment variable (or command-line option). If CPython adopts some form of the GIL changes, I’d expect this runtime control to be useful for at least a few releases to address flag day issues.
This doesn't say "GILectomy will forever remain an opt-in".
https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD...
In particular, the design document's observation that "some C API extensions need modifications to protect global data structures that were previously protected by the GIL" serves as a direct confirmation that this change was not able to avoid breaking at least some multithreaded code that relies on the GIL to provide thread safety.
Yes, C extensions that count on the GIL for handling their own internal locking will obviously need updates.
1. Give the performance-benefits of GIL removal for multi-threaded code.
2. Not hurt the performance of single-threaded code, hopefully improving it too.
3. Not break any pure Python code that relies on the semantics of the GIL.
4. Minimize impact to extensions.
If you're commenting here that this is going to break things w/o having read the design doc, you're contributing FUD. It's disappointing to see the discussion go this way.
Here's Sam's design doc:
https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD...
Some highlights:
The project aims for a concurrency model that matches the threads + shared memory model implemented in common operating systems and in programming languages like Java, C++, and Swift. We also aim for similar safety guarantees to Java and Go -- the language doesn’t prevent data races, but data races in user code do not corrupt the VM state. (Notably, Swift and C++ do not have this property: data races in those languages may corrupt VM state leading to segfaults). This strikes a balance between safety and performance. Weaker guarantees, like in Swift and C++, can make debugging difficult, while much stronger guarantees can limit flexibility or hamper performance.
Compatibility:
The vast majority of Python programs should not require any changes to Python code to continue working. Most of the issues I have encountered so far are due to being based on 3.9.0a3 and libraries using features from the 3.9 final release. A few of the issues have been due to changes to code object internals or the bytecode format, but so far I have been able to address those by making changes to the interpreter to improve backwards compatibility.
All extensions that use the Python C API will require re-compilation, even the few that are limited to the stable ABI. (The reference counting changes do not preserve the stable ABI).
To compile successfully, some C API extensions will need minor modifications, typically replacement of direct access of PyObject reference count fields with the Py_REFCNT macro.
To run successfully, some C API extensions need modifications to protect global data structures in C code that were previously protected by the GIL. For an example, see these two patches to NumPy.
It's a classic case of "you're doing it wrong": Python supports concurrency in several useful ways, and specifically does NOT support it in one particular very useful way. If you have somehow arranged to bang your head on that one specific thing that Python doesn't do, and you cannot figure out how to not bang your head on the GIL, then for goodness' sake use Go or Erlang or something that DOES do the one thing you can't live without.
- - - -
Don't get me wrong. If this succeeds it will be a great thing. This effort seems well-thought out, and I wish Sam Gross luck and success.
https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD...
Because, you know, they make a valid (flamebatish) point, the ‘easiest’ way to get performance out of python is to wrap the critical code in another language as an extension. Removing the GIL isn’t going to change this no matter how careful it is designed.
I read the document. Sam Gross seems to have a clear grasp on the risks associated with this kind of project (I would quote the relevant part, but google docs is a clown shoe.)
One the one hand I wish him luck. If he pulls it off he'll be a hero. On the other hand this seems like a huge "turd polishing" for a problem that one only encounters if one is doing something ill-advised. The GIL just isn't that big of a problem in practice. I've been writing Python for going on twenty years without ever encountering a single problem due to the GIL.
I haven't encountered any difficulties with the GIL either, but that doesn't mean it's not a problem.
I really do hope Gross is successful. It does seem like he's "got an angle", so to speak, on the problem. At the very least, he's coming into it well-informed with his eyes open. He wants to try excising the GIL (and FB is apparently footing the bill to boot) so it's stupid to object, eh? Who cares if he wastes his time? He's not hurting anything. And maybe the horse will learn to sing. :)
The FUD comments came when I was on my phone and in https://xkcd.com/386/ mode. I was just disappointed seeing so many comments that were, well, FUD. FUD may be terse, but I don't think it's strong language.
There's a lot of misunderstanding around the GIL in general.
I too hope he is successful.
Moreover, to the point about writing performance-critical sections in $NOT_PYTHON, as Sam Gross explains in the nogil design doc, things aren't always so straightforward for scientific computing:
> Calling into Python from C requires acquiring the GIL -- even short snippets of Python code can inhibit scaling. Addressing this issue sometimes requires rewriting large portions of Python code in C/C++ to actually achieve the desired parallelism. For example, PyTorch rewrote all of the core automatic differentiation from Python to C++ to avoid acquiring the GIL in the backward pass.
https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD...
This seems like a key consideration for the long-term success of nogil and CPython. The Python ecosystem has survived and recovered from past forks but it would good to avoid forks altogether if possible, especially if nogil proves to be stable and performant enough for prime time. At the least we should try to keep any forks short-term, e.g. starting with a compiler flag with a timeline for shifting to a runtime flag. (These are just my thoughts, take everything with a pinch of salt.)
Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you make incompatible API changes
--------
Considering semver and that this is a COMPATIBLE API change, it does not warrant an increment to 4.x
1. python3.x nogil as a compiler flag
2. python3.y nogil as a runtime flag
3. python4.0 nogil as the default (if we’re really ready for it)
I can’t really further consider this or get excited about it unless I understand how it’s going to work and how it addresses previous issues with removimg the Gil.
There are still many cases when data scientists don't have a C extension and need to run pure Python functions in parallel, and the GIL makes this an order of magnitude harder (and sometimes slower).
I think the core Python devs have gotten rather too conservative in the language's old age.
It makes some operations atomic which is quite relevant.
Sure, but any call to a native function which doesn’t release the gil (which is most of them at least for the builtins) is currently atomic so something like dict.setdefault.
But the point is currently they do work, they are thread-safe and they are correct in the strictest sense of the word, per-FAQ: https://docs.python.org/3/faq/library.html#what-kinds-of-glo...
So as the original commenter notes:
> wouldn't removing the GIL cause all sorts of sudden race conditions and regressions in existing python code?
R is a interactive statistical programming language that acts as a frontend for more performance languages. AFAIK interactivity is not the strongest points of Julia at the moment.
Not sure what you mean. Julia has the exact same notebook environment (Jupyter) as R and Python. Fun fact: the “Ju” in Jupyter stands for “Julia” (the “pyt” and “r” stand for what you think).
For example, take a look at the reactive Julia notebook, Pluto.jl: https://plutojl.org/plutocon2021
I mostly do scientific computing but I really enjoy the toolchain of an extremely popular multi-purpose programming language (Python) + C extensions where necessary for speed. The reason is that many tasks these days are not just pure number crunching. If I need to start up a quick web server, do some web site scraping, basically anything can be done easily from Python. It's unlikely Julia will ever be able to catch-up in these areas.
https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD...
docker run -it nogil/python
Once the extensions and eco-system in general catches-up the flag can be a runtime one (default = OFF)
It's hard to put to words what the GIL does for Python. I think the best I can do is to point to Rust and ask you to consider what Rust's number one claim to fame is? A more than significant part of Rust's bible ("the book") is about memory ownership. Python's GIL lets me be completely ignorant of these issues and all my brain cells just focus on the problem at hand.
My long term livelihood and sanity is at stake here and I really appreciate that some of core maintainers of Python are taking their time and thinking deeply about the implications of removing the GIL.
Python with the big bad GIL has become the most popular language in recent years. And it's gotten there slowly and steadily over 20 years. The most popular language just so happens to be the only (?) language that is single threaded.
The lack of first class FP, beyond the type checking stuff, may actually at root be part of the problem: if Python had somehow become the FP rich language it said "nope, different model" -then parallelism and GIL would be a totally different argument.
Maybe I was mis-informed. FP interests me but I am not rich in experience. Maybe its an oversold idea?
J might be called functional--it does permit global side effects, but discourages them and encourages immutable patterns.
We are currently working on threading support. There is no GIL.
More generally, you don’t have immutability / value semantics by default.
In this context it seems a bit misleading to care about function call overhead (except for TCO) because Python is so slow overall. I'm usually seeing 30x when translating to C. (For pure Python of course, not PyPy or code making good use of numpy.)
To continue that example, Python list comprehensions are not very powerful compared to that feature in FP languages, since Python is not expression orientated.
I would classify Python as imperative with FP and OOP elements.
Python is of course a slow language. My point is that writing in a functional style will give you slow code even by Python standards! Good FP languages assume you will write in that style and optimise for that use-case.
that said, I'd like to see the GIL removed.