never ever had a problem with GIL in python, always found a workaround just by spinning up ThreadPool or ProcessPool, and used async libraries when needed.
is there any use case of No-GIL which is not solved by multiprocessing ?
I thought Single threaded execution without overhead for concurrency primitives is the best way to high performance computing (as demonstrated by LMAX Disruptor)
No-GIL multi-threading is multi-core, though difficult to use. I don't know the Python implementation but shared memory should be faster than using multiprocessing.
That said, when designing a system from scratch, I completely agree with you that for almost almost almost all Python use cases, threads should never be touched and asyncio/multiprocessing is the way to go instead. Most Python programs that need fast multi-threading instead should not have been written in Python. Still, we're here now and people did write CPU-intensive code in Python for one reason or another, so no-GIL is practical.
In these threads, I also always see a lot of people who simply aren't aware of asyncio/multiprocessing. I assume these are also a significant share of people asking for no-GIL, though probably not the ones pushing the change in the committee.
This design scales much better for systems that will eventually overgrow one big machine. the No-GIL pytohn will be of no use, when you need to deploy your app across 100s machines.
I understand people want to take advantage of all cores etc, but at large scale - you will eventually need to split computation across machines and will resort to in-memory cache/queue anyways - so better just architect your system since day0
probably a way more problematic effort than no gil python tough and it might fail
More machines is a multiplier. Instead of underutilising 1 machine, you'll be underutilising 100s of machines. More incentive to remove locking.
This will just allow people to waste even more compute resources to get somewhat quick results from Python instead of doing the right thing.
From an ecological viewpoint this PEP will be disastrous and keep entire power plants online.
The whole point of using languages like python, js etc. is to not worry about whole classes of problems and intricacies.
There are modern languages with great tooling like Go, Rust or Clojure that have excellent support for safe concurrency and parallel execution etc.
And there are reliable, well established languages like Java/C# that have been doing this forever.
The seamingly endless attempt to become more likeable to yet another subgroup of needs is not a good development. It started with static typing, it continued with async, and now we have free threading as the final draw.
If someone really needs several threads of pure python being interpreted, something is afoul imo.
IIRC some of the proposals around removing the GIL in the past have actually suggested that the asyncio paradigm could become multithreaded for parallelism.
Tons:
- A web server that responds using shared state to multiple clients at the same time.
- multiprocessing uses pickle to send and receive data, which is a massive overhead in terms of performance. Let’s say you want to do some parallel computations on a data structure, and said structure is 1GB in memory. Multiprocessing won’t be able to deal with it with good performance.
- Another consequence of using pickle is that you can’t share all types of objects. To make matters worse, errors due to unpickable objects are really hard to debug in any non-trivial data structure. This means that sharing certain objects (specially those created by native libraries) can be impossible.
- Any processing where state needs to be shared during process execution is really hard to do with the multiprocessing module. For example, the Prometheus exporter for Flask, which only outputs some basic stats for response times/paths/etc, needs a weird hack with a temporary directory if you want to collect stats for all processes.
I could go on, honestly. But the GIL is a massive problem when trying to do parallelism in Python.
> Manuel Kroiss, software engineer at DeepMind on the reinforcement learning team, describes how the bottlenecks posed by the GIL lead to rewriting Python codebases in C++, making the code less accessible:
> "We frequently battle issues with the Python GIL at DeepMind. In many of our applications, we would like to run on the order of 50-100 threads per process. However, we often see that even with fewer than 10 threads the GIL becomes the bottleneck. To work around this problem, we sometimes use subprocesses, but in many cases the inter-process communication becomes too big of an overhead. To deal with the GIL, we usually end up translating large parts of our Python codebase into C++. This is undesirable because it makes the code less accessible to researchers."
For average usage like web apps, no-GIL can be solved by multiprocessing. But for AI workloads at huge scale like Google and DeepMind, the GIL really does limit their usage of Python (hence the need to translate to C++). This is also why Meta are willing to commit three engineer-years to making this happen: https://news.ycombinator.com/item?id=36643670
Probably was easier to have one crack team of software developers write something new which could interpret all of the existing codebase, than it was to lead a widespread conversion of all of that code into faster languages.
ie. not everyone's a senior dev. There's reams more junior devs coming from bootcamps and such, computer science grads, etc, who can grok "scripting" languages like python and JS and Java much easier than they can pick up C++SIGSEGV or Rust algebraic data type smart pointer closure trait object macros.
Think how much of the world is boring "business logic" and it makes more sense - focus efforts making the [on the surface] simple, widespread, generalist, scripting languages, faster - we've seen it with Python, we've seen it with JS (node), we've seen it with Java.
Given how big the slow languages are, it makes lots of sense to save their CPU cycles compared to trying to hire from a much smaller pool of "competent at lower level programming" devs.
RE "scripting" vs "compiled" - I'm probably using the semantics wrong :p
To me, "scripting" is more like... Rexxx, or Lua, or Bash. Stuff that's turing complete but more restricted in how you can express things in the code, or sandboxed (designed to be embedded). Python may have started off designed to be embedded as a scripting language, but these days it's a very very general purpose _predominantly interpreted_ language, considering where it's used and the libraries it has. It's not just used inside of Blender or OpenResty for example.
I'd argue the same about Perl and PHP. If people (psychopaths) are content with using PHP-Gtk to make _desktop_ apps, does it count as scripting language the same way "Lua embedded as a way to make Source Engine entities interactive" is a scripting language? :p
> FAANG companies such as Facebook can pick and choose from the highest caliber developers.
Sure - they have lots of money. They're also very, very big companies, with offices all over the world. Look at the sheer amount of people they hired which they backtracked on later "oops, we hired too many of you, haha, sorry! layoff time!". It doesn't take 10+ years of experience to come fresh from a coding bootcamp, complete the Google code test and become a Noogler in the "Wear OS performance metrics" team writing boilerplate AsyncTasks that call UrlRequests all day every day. Plus on the positive side they encourage people to join through undergrad / new grad schemes. Like, isn't there a whole thing about people _starting_ their tech careers in FAANG corps?
That's why many code bases are a mess because everything is dictated by almost teenagers (500 per code base ...) who rewrite everything daily.
1. Because asyncio is completely useless when the problem is CPU bound, as the event loop still runs only on a single core. As the name implies, it is really only helpful when problems are IO bound.
2. Because sharing data between multiple processes is a giant PITA. Controlling data AND orchestrating processes is an even bigger pain.
3. Processes are expensive, and due to the aforementioned pain of sharing data, greenlets are not really a viable solutions.
The number crunching does, but distributing the workload, receiving results, storing and retreiving data, etc. doesn't. And these are huge losses in performance that could be avoided if we could parallelise them.
It's pretty nice to be able to just throw cores at a problem during prototyping phases, compared to throwing developer time to write native code.
This is a real problem-- which is why there's a whole lot of numpy/scipy maintainers in favor of the change.
Anything that benefits from both parallelism and replacing IPC overhead with shared data between parallel tasks.
I dont believe python will ever have atomic operations, even if it had - they still incur significant overhead for concurrency control.
sharing state between threads is such a narow niche use case, this pattern is practically solved by memcached/redis for larger scale python based systems
at which point you lose benefit of no-GIL, because you now have to introduce redis and kafka into the system
Why?
Yeah, if you are building a system that is, say, serving web requests, and have an internet scale potential market, success might mean that.
Not every system works that way. A simulation system with defined parameters doesn’t grow in scale if it becomes more popular, you just have more people running isolated instances that don’t depend on each other. Plenty of other applications scale that way rather than the “SaaS that serves ever more clients” way.
Still lots of complexity and awkwardness that can be avoided if you can do threading instead of processes. Like Promotheus scraping from a non-webserver python app is a pita, as you need a new process and lots of communication, vs just plug and play as in other languages.
Or just the insane resource usage. Had a java app serving multiple orders of magnitude more customers running on a few containers. Our current python app needs multiple deployments with different entry points, and about 15x the amount of containers.
That policy is over.
As the last years have shown, no alternative implementation can get off the ground due to C extensions and compatibility concerns, and CPython is now relied on for many large applications. It no longer makes sense to prioritise a simple implementation over performance.
Finely-grained locks are useful. Even when you end up scaling between machines, it can be useful to have many threads in one memory space to maximize what you get out of one machine.
We're moving up to hundreds of cores; Python often being stuck only being able to use a couple while tightly coupling state has been unfortunate.
Building applications that run on multiple machines is at least one order of magnitude more complex and thus slower (in development velocity), so needlessly building an application to work distributedly is just bad engineering.
And "one beefy machine" has a very high limit, so by the time you actually outgrow it you usually have tons of resources available to help rewrite things.
You might think that you can simply spin M processes per machine instead but now you have N*M servers instead of N servers that are M times faster. In many cases this means you have significantly higher overheads: slower startups, a lot more RAM usage, more network IO etc.
Outside of a few embarrassingly parallel problem, two-level parallelism is usually the highest performance approach.
Yes, but there are plenty of problems where that is more development overhead, but less runtime overhead than IPC.
It is the norm. „Kafka scale“ problems are not the norm.
Outside of python, no it really isn't, it's the norm.
And even within python itself: That redis connection IS shared state.
I am thinking about multithreading every day to try make it easier to use. I journal about it in my ideas journals.
Threads let you use the same unified abstraction for parallelization and concurrency. They also make it easier to share state with parallelization (no need to go out of your way to do it) at the cost of requiring you to think about and implement thread safety when you do so.
Also, with no-GIL + threads the computational costs of creating and maintaining a parallel execution is much less vs multiprocessing. And data sharing and synchronization are less expensive.
What LMAX is doing is really just an overhyped way to speed up producer-consumer models. It might apply to your use case but it’s not the only reason you’d use parallelism or concurrency. I don’t even understand why they are claiming it to be an innovation when it’s just using a LockFreeQueue implementation within a pre-allocated arena? You also can’t synchronize with their implementation, which sometimes you really need to do. Not a silver bullet
1. random jumps in memory and branch misses
2. L1/L2 cache flush
3. context switch cost
4. concurrency locks cost
1. pre-allocated arena ensures cache locality of operations
2. we dont jump form one sector of memory into another. Algorithm more resembles linear scanning of working memory set, and mostly within L1/L2 cache
3. no context switches, no cache flushes
4. no concurrency control costsLet’s say you did try to implement that in LMAX. It’s common for consumers/“workers”/what have you to require synchronizations amongst themselves, for example if they are operating on a shared k:v store of strings (operating an in-memory db let’s say). You can’t do atomic reads or writes on the thing so you need a locking mechanism; under LMAX you’d have to introduce another layer of producers to control reads and writes and then have another layer of consumers afterwards to handle the rest of your “consumer flow”, or wait in the original consumer thread for the producer to complete, which starts getting very wasteful and is certainly no better than a typical regular locks and context switching.
Again, this is not even a new thing. Lock free queues and “local atomic concurrent pub-sub” have existed for a long time - we have an implementation where I work. It’s not a perfect model even for where that concurrency pattern is wholly sufficient for what you’re doing either - the performance boost from the cache and context switching improvements have to be greater than the slack (in cost or throughput) introduced from producers or consumers sitting idle waiting for upstream data.
Also, context switches/cache invalidation/concurrency overhead can be avoided or at least greatly reduced by smart userspace scheduling a la fibers. With hand tuning it can potentially be completely eliminated (you can control which concurrency units to collocate on a thread and resume concurrency units/threads immediately after their waiting locks free) which is basically the same idea as LMAX. The problem of course, like with LMAX, is that doesn’t generalize.
> Python’s global interpreter lock makes it difficult to use modern multi-core CPUs efficiently for many scientific and numeric computing applications. Heinrich Kuttler, Manuel Kroiss, and Paweł Jurgielewicz found that multi-threaded implementations in Python did not scale well for their tasks and that using multiple processes was not a suitable alternative.
> The scaling bottlenecks are not solely in core numeric tasks. Both Zachary DeVito and Paweł Jurgielewicz described challenges with coordination and communication in Python.
> Olivier Grisel, Ralf Gommers, and Zachary DeVito described how current workarounds for the GIL are “complex to maintain” and cause “lower developer productivity.” The GIL makes it more difficult to develop and maintain scientific and numeric computing libraries as well leading to library designs that are more difficult to use.
Let's hope the change is not this badly mercenary.
I mean a situation like where in threaded context there would be code like:
def determine_quest_latency(quest_name: str) -> int:
return other_thread.wait_sync_job(lambda context: context.ping_quest(quest_name))
I believe the answer is "you don't", but passing functions in messages is a highly convenient way to structure code in a way that local decisions can stay local, instead of being interspersed around the codebase.
If you really want to pass lambdas, you can use a third party library to pickle them
https://github.com/cloudpipe/cloudpickle
Yes, this is not great.
You can have shared-memory parallelism with near-zero synchronization overhead. Rust's rayon is an example. Take a big vector, chunk it into a few blocks, distribute the blocks across threads, let them work on it and then merge the results. Since the chunks are independent you don't need to lock accesses. The only cost you're paying is sending tasks across work queues. But that's still much cheaper than spawning a new process.
But I have the same question as you have if we add another coming concurrency model: SubinterpreterThreadPool, which will be possible with the per-interpreter GIL in python 3.12 and later.
That's another new model that is already confirmed to be coming: interpreters (isolated from each other) in the same process, that can run with each their own GIL.
with PoC of the parallelization code being developed here https://github.com/jsbueno/extrainterpreters
Multithreading is more efficient but more difficult to work with.
You share the same address space in threads, so you can communicate any amount of data between threads instantly within a lock. The same cannot be said for network traffic or OS pipes or multiprocessing.
Multiprocessing uses pickle to serialize your data and deserialize it in the other python interpreter.
If you start a Python Thread, you're still single threaded due to the GIL.
Multiprocessing does the parallel computation pretty well as long as the granularity is not too small. When smaller chunks are needed most of the time that's something better done from an extension.
For creating stateless, mostly IO bound, servers, it's great. Try to squeeze in performance and it all starts to fall apart.