Why Not Python - the GIL hinders concurrency
chrisstucchio.com
chrisstucchio.com
A few remarks: I would qualify the GIL as a tradeoff rather than a mistake. You loose CPU-bound parallelism with threading, but you gain easy to write C extensions (well, relatively speaking). I think this point is critical to the existence of something like NumPy (which is unrivalled in general PL AFAIK).
If you need to share a lot of data (a big numpy array), then you can use mmap arrays, there is no serializing/desarializing cost, and that's efficient. See for example some presentations by O. Grisel from scikits-learn (https://www.youtube.com/user/EnthoughtMedia/ -- disclaimer, I work for Enthought, but all those video happened at scipy 2013).
IMO, the only significant niche where this is an issue is reaching peak performance on a single machine, but python is not really appropriate anyway -- that's where you need fortran/C/C++, and even scala/java/haskell are quite unheard off there.
The existence of GIL may even indirectly speed up numerical Python code. Numpy and cython are significantly faster than N pure python threads, and the GIL encourages their development.
Everything else being equal, I would rather not have a GIL. I don't know any efficient runtime which uses reference counting and has no GIL, and integrating C with garbage collection is generally hard. That's one of the main reason why integrating C/etc.. with JNI or Matlab is a nightmare. The only one that managed to reduce the impedence mismatch and that I am aware of is Lua.
And, from the speed and design of CPython, we know that they are not V8-level or JVM-level competent interpreter/VM writers.
They went about it by adding mutexes around the reference counters (as opposed to using a separate mark & sweep GC, which has it's own downsides, as displayed by the JVM). This resulted in somewhat increased performance of multi-threaded applications, at the cost of single threaded applications.
> And, from the speed and design of CPython, we know that they are not V8-level or JVM-level competent interpreter/VM writers.
The biggest problem that V8/JVM (i.e. JIT) style interpreters have, when interpreting the Python language, is that JIT compilation is not really possible (or rather, exceptionally difficult). It's simply not possible to say that any given object in Python will remain it's currently defined type, or that an underlying method will not change meaning as execution continues.
one + two
PyPy is currently trying to come up with a JIT compiler, and they're having remarkably good success. Rather than ragging on Python, try supporting them instead.
The same thing holds for JS. And yet, they have tons of heuristics to counter it.
Not to mention that most objects really don't change anyway.
>one + two can (and does) change it's meaning dynamically during the execution of a Python script, and simply optimizing away the __add__ lookups will bite you in the ass.
That's not the case for over 90% of the code I've seen in Python. Even if it DOES use operator overloading.
I'm extremely interested in your thoughts on this subject as a NumPy developer. To what extent do you think C interop designs like Python CFFI (which mimics LuaJIT's FFI) will make this an unimpressive accomplishment?
As a former heavy user of Python for scientific programming (for easy access to C libraries) I have moved everything I do to LuaJIT and C. The ease of writing a LuaJIT FFI (or Python CFFI binding) is at least an order of magnitude greater than that of writing PyObject* style bindings.
Do you think it's possible (or likely) that a mature CFFI will make this tradeoff you describe seem no-longer-positive?
Added in Edit: I should point out that multicore patterns in Lua are already very different from those in Python, e.g. running a Lua VM per core and communicating through shared memory. So I realize that the existence of CFFI won't map one-to-one onto changes in Python multicore design, at least in the short term.
Unfortunately, I think that for python, it is too late to have a better C API: there is so much legacy that depends on python C API details that moving away from that would take almost as much manpower as rewriting it to a different language.
Look at pypy: even though it is 5x times faster on average than python, I don't see people flying from cpython to pypy. 5x speedup is more than what you can hope on todays' CPU with near perfect scaling with # cores. This tells me that no-GIL python, to be successful, would need to have a much lower barrier to entry than pypy. Maybe the future is closer to having hooks to 'escape' python for the numerical stuff: numba and the numerous similar projects jitting a subset of python, or interoperating with some other language more amenable to optimization (e.g. Julia: http://www.youtube.com/watch?v=Eb8CMuNKdJ0)
I am quite interested in Lua-like design, though: one of my pet project around numpy would be to refactor it internally to use something closer to how lua works, exposing numpy to cpython only at the outer layer.
I think we'll have to agree to disagree on this. To quote the CFFI project description [0]: the goal is to be able to seamlessly call C from Python without having to learn a third language/API. Cython, Ctypes, and CPython's C API all fail to meet this criterion, and in my opinion do so by a large margin. Let's also not forget the fact that Cython requires a compilation step not only for the bound C code but also for the portion of the "Python" code that interfaces with it.
Compare that (or writing CPython C API manipulation code) with the following:
>>> from cffi import FFI
>>> ffi = FFI()
>>> ffi.cdef("int printf(const char *format, ...);")
>>> C = ffi.dlopen(None) # loads the entire C namespace
>>> arg = ffi.new("char[]", "world") # char arg[] = "world";
>>> C.printf("hi there, %s!\n", arg) # call printf
hi there, world!
local ffi = require "ffi"
ffi.cdef [[ int printf(const char*, ...); ]]
ffi.C.printf("hi there, %s!\n", "world")
Anyway, leaving personal taste aside for a moment, let me rephrase the original question that sparked my interest in your opinion: in a world with a mature CFFI option providing the above binding capabilities do you think that easy C extensions are still a worthwhile tradeoff for the GIL? (FWIW I find the point to be quite compelling for why Python had a GIL to begin with in the late 90s, but I'm suspecting it's less of an intrinsic design tradeoff and more historical cruft as the PyPy and CFFI people roll out their work).
Wrapping things like crypto, file format and co is relatively trivial, because not much crosses the language barrier. Now, when you need to handle non native object life cycle, that's another matter, and I don't see how cffi makes it any easier than cython
I will close by saying that, while I agree with you that CFFI doesn't make the problem of writing memory-safe C extensions to the Python VM any easier, I also believe that almost no one (aside from people like Numpy developers) actually need to write a Python extension. They just write extensions (or use Cython, or Ctypes) because that's "the way" to call C from Python.
In my personal experience 90+% of the Python+C problems in the world are just about calling an existing piece of C code and maybe mapping some structs or arrays into Python types; these workflows rarely involve instantiating complex objects on the Python side. For my work habits neither CPython, Ctypes, nor Cython are optimized for this extremely common use case. CFFI solves these problems for me better than any of those alternatives, and if someone offered me a Python 4.0 with CFFI and no GIL at the cost of harder-to-use extension mechanisms I'd jump on it; I suspect that a significant portion of the people who do Python/C interop work would feel similarly. I also do realize that such a change could make projects like Numpy harder to build.
Now I agree it might not be convenient, but that's a matter of libraries. This post would be much more constructive if it was speculation on what such a library could look like, instead of pretending that concurrency "doesn't work" in single-threaded Unix processes.
1. http://docs.python.org/2/library/multiprocessing.html#module...
I'm sure it's possible to build an IPC garbage collector and enable everything described. But to describe it as "not convenient" is perhaps a small understatement.
A shared memory hash table actually seems like a really useful thing to eventually make its way into the multiprocessing module.
For the event example, parsing the data structures into shared memory and then accessing from there would remove the parsing and memory overhead he complains about. I don't really want to dig in to the standards he cites and see how difficult that solution would be, but again, I can't see a reason why it isn't possible.
At any rate, in both cases, the solution is definitely doable; there's nothing inherent in python's concurrency model that "prevents you from properly making use of modern multicore hardware"
Side note: shared memory access in modern multicore hardware may not get you the performance gains you would initially think. Especially for write-heavy workloads, cache invalidation is a huge problem.
A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future.
I agree; however, this problem could still be bootstrapped to each individual process's GC. The parser sends the shared memory id to all worker processes, and the shared memory representation contains a reference counter. Each object decrements the reference counter in __del__; when the reference counter reaches zero, the memory is unmapped and reclaimed by the kernel. This would probably cause major fragmentation issues with small objects, so pooling might be necessary.
This sounds like a job that could be abstracted by a library.
> A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future.
I won't argue that shared memory hash tables (and concurrent data structures in general) are easy to implement. But that leads back to my original point; that kind of code should be in a library.
Admittedly, Python today lacks libraries providing concurrent data structures. And it lacks modules that handle concurrent memory management. That doesn't mean there's something inherent in its design that prevents such libraries from existing.
However, I think that the threaded model is in general easier to work with, rather than shared memory. That said, this might be offset by python itself being easier to work with than another language.
Also, separating work into multiple processes gives in general better security - one crashing worker won't impact the others. Then again, you won't take advantage of hyper-threading features of the CPU.
In the end, everybody needs to consider their personal needs and make a tradeoff.
The thing is, threaded models of programming in other languages already use the "shared memory" model, they just make it more implicit. Which leads to tons of problems for inexperienced developers who don't really understand concurrency, or the performance problems that shared memory can bring in modern processors.
> In the end, everybody needs to consider their personal needs and make a tradeoff.
Agreed.
Well, sorta. You're still incurring the mighty overhead of multiple processes and all that it entails if you want to dynamically resize the set of workers. One of the reasons that Scala, Erlang and Clojure have it so good on the concurrency AND parallelism front is that from their perspectively, these things are both cheap in terms of the overhead and the syntax.
And of course there's locality to consider.
One of the lesser-appreciated facts about the GIL is that it is an implementation detail of CPython. That is, it is entirely possible to implement a Python that does not have process-level globals and C statics. There is no structural reason why a C or C++ implementation of Python needs to have a GIL. It's just a legacy of the implementation that has stayed around for a long, long time.
For instance, Trent Nelson has done some work to show that you can move to thread-local storage for the free lists and whatnot, and get nice multicore scaling, even with the existing CPython codebase[1]. There are still concurrency primitives that the language would need to offer at the C API level to manage the merger of these thread-local objects, but it's a whole lot better than only being able to use a single core in modern days.
Fortunately I mostly get to work in the scientific field with (mostly) data parallel problems.
[1] https://speakerdeck.com/trent/parallelizing-the-python-inter...
But fundamentally, the GIL prevents Python from being used as a systems language.
As such, I'd only be willing to agree with the author's statement for a very specific subset of systems programming.
For example, I've worked on a packaging system written in Python for the last five years or so. The package system is primarily I/O-bound the vast majority of the time (waiting on disks or network), and almost all significant performance wins (some as much as 80-90%) have come from algorithmic improvements rather than rewriting portions in C (very little is written in C).
As one of my colleagues is fond of saying (paraphrasing), "doing less work is the best way to go faster".
It also ignores the fact that depending on the problem space involved, there may be readily available solutions that provide excellent performance that don't involve threading (e.g. the multiprocessing module, shared memory, etc.).
Also, as others have mentioned. Two processes can share memory (at least on Linux) by explicitly using the shared memory system calls. However, this is complicated in Python by the need for reference counting of objects created in the shared memory region.
I do however feel the need to make a couple clarifications:
If you fork a process, you don't necessarily duplicate memory. Yay for COW. Fork twice and you've got fanout with very little extra memory. Threading works fantastically for I/O. C libraries can release the GIL during processing. So if you have to do something computationally expensive, you can let another python thread run.
I rarely feel restricted by python, and when I do I usually find that other aspects make up for it. I find it makes a decent systems programming language (although not as good as C). The os library and the ctypes library give you the capability to do most (all?) system level tasks.
I tend to prefer go over python when library support isn't an issue, but that's more due to it's type system and channel primitives. As someone who's pushed python to it's limits as a systems programming language I'm pretty comfortable dropping down to C isn't often called for, and writing in java/scala isn't ever required.
I'm not a ruby expert but by understanding is that basically they've moved the refcount field out of the struct and out of the memory page. It would be nice if python did something like this.
[edit] My summary of what ruby does is totally wrong and while this is interesting and applies to memory managment, doesn't necessarily apply to refcounting.
The golang community has lectured about this at length: http://blog.golang.org/concurrency-is-not-parallelism
A great example of concurrency is a gui application. Without threads, any blocking operations will make your application unresponsive. You need multiple threads so that if one thread is blocking on a syscall, the other is available to execute gui actions. This works in python, because the GIL only applies to actually executing python threads.
As far as writing general-purpose applications in python goes, most don't hammer all cores with work. In fact, they usually only require I/O concurrency, so python works as well as any other language.
Why are you doing compute intensive work in Python? Python is not well optimized for doing compute intensive work. It's made some strides over the years, being able to do limited bytecode optimization and multiprocessing, but it's not a high performance computation language.
However, it is a fantastic glue language. Write your compute intensive portions in C, and use Python to glue the portions of C together. You can even do your thread generation in Python, release the GIL during your C execution, and you don't even have to worry about the complicated process of spinning up threads in your C.
In other words, use the language like it was designed to be used.
Of course, not releasing the GIL eases interaction with Python level variables, so that might be a tradeoff they decided is worth losing threading for.
But please don't blame Python for the choices made by a library developer.
Um, what? They do not hold on to the GIL when doing compute intensive operations.
This advice is so often given by people, that I suspect that most people don't know what they are talking about, because in fact most people never do that, being unable to drop to C. I dare you to show me a piece of "compute intensive" portion that you optimized with C.
I've worked with Python for about 3 years. After struggling with it to stretch the boundaries of what it could do, I finally gave up in frustration and the final solution was to find a less limiting environment. Worked great thus far and I don't miss anything about Python.
I acknowledge this is very anecdotal... but my (previous) labs observed a tradeoff in developer productivity in Python vs C, and deemed Python worthwhile (avoiding classic premature optimization and whatnot).
Honestly, it's not that hard to do - mostly a bit of wrapper code to announce to Python what methods you're exposing, and doing coercion from Python types to C types. The hardest part for me was identifying when to increment and decrement reference counters to variables being passed in to me.
Because I'd rather have a unified code base, all higher level, than some BS mix of Python and C or other stuff that I'd have to support.
>Python is not well optimized for doing compute intensive work
That's a rather circular argument, when the topic under discussion is why Python is not well optimised for doing compute intensive work.
What higher level language offers you the high performance computing you desire?
> That's a rather circular argument, when the topic under discussion is why Python is not well optimised for doing compute intensive work.
That's not the original post I read - it was berating Python for not being a high performance computing language. To which I reply: Of course it's not. It's the wrong tool. Expecting a mature high level glue language to also be exceptionally high performance... that language doesn't exist yet. Go is a good step along that route, but it's own hacky interoperability with C and lack of maturity creates its own problems.
Scala and Haskell both strike a nice balance for my uses. They aren't quite as fast as C, but give me decent concurrency primitives and performance.
(You might be surprised to discover that my use cases are primarily the realtime event processing systems described in the article.)
The major pain problem with other high level languages is the lack of a large library ecosystem (to replace NumPy et al).
As for which high level languages that beat the pants out of Python for high performance computing, there are plenty.
Julia for one. Haskell for another. Scala. Go. LuaJIT It's a long list. Heck, even V8 is miles ahead.
this is incorrect. The processes coordinate on a lock held in redis itself.
This solution is available right now using the Redis backend in dogpile.cache (of which I am the author): https://dogpilecache.readthedocs.org/en/latest/usage.html
http://www.dr-josiah.com/2012/01/creating-lock-with-redis.ht...
[link redacted]
When multiple processes ask this for the same file one of them downloads it and the others wait for it to finish.
Multiprocessing is the way to do parallelism. Deviating from that should be an exception -- for example, shared memory maps could be used to transfer select data objects instantaneously between the processes instead of serializing/deserializing over a pipe, and only those while still retaining separate process images. Threads were practically invented as a compensation for systems with heavy process image overhead.
I think Python is very Unix in this regard. And that's not a bad thing per se. Unix and Linux can do multiprocessing very efficiently.
Unless you have a language that doesn't break down when you use threads. Threads are unquestionably easier and better to use for multiprocessing when they work without the tools you are using breaking that.
Nowadays I use Scala/Akka for concurrent event processing. As far as code organization goes it is a lot like multiprocessing. Just s/actor/process/. But unlike unix style forking/IPC, I don't have to waste memory/cpu cycles serializing/deserializing immutable messages just so another process can have a duplicated copy of it.
Why use threads?
Noone does parallel compute on CPUs these days, not since GPGPUs rocked up almost 5 years ago (and we often use python as the host language, thanks pyCuda!)
Parallel IO then? Well, except that async IO is often far more resource efficient (at the cost of complexity though).
Threading is dead(-ish) because its hard to write, hard to test and expensive to get right.
Concurrency in python is very much alive though.
I want to live in your world where all you are processing is vectors and FFTs in parallel on GPUs and not doing real work (accessing databases, processing data from sockets, etc).
Threading is not dead. It's only crippled in python so everyone wants to invent ways of saying it is dead.
Threading being hard to write is also a fallacy. I use thread backed dispatch queues which make concurrency simple in my language of choice right now. Threading like that is easy thanks to closures and a good design patterns. My apps are entirely async and run heavily parallel and it's easy to maintain and write using that.
For all your IO cases, and all your cases are IO, would you, and future maintainers of your code, not be better served with simpler abstractions which permit scaling past a single host?
But referring to the IO side debate, the current design of many of the libraries that you call in the C world are often inheirtly blocking. 'gethostname' for example is a blocking call. There is no async version of it. To use them without contention on your single threaded application, you have to call them from worker threads.
The common pattern is to spin up a thread to call it and do work on it. It's easier often to have your workers be thread bound like that to simplify your code and only lock shared resources when you need them. I can also make a massively async version of all my code that handles everything using async methods and in many cases this is better but it's harder to write and not always an option. Something I have to deal with daily because I run into the C10K problem all the time at work (http://en.wikipedia.org/wiki/C10k_problem).
Even in the async model though I still want to be running code in parallel and I would still rather build that model up with thread powering it and not multiple processes and shared memory.
The actual computation, the bit that was previously multi threaded (or more commonly, multi process) on a CPU, now lives on a GPU. I'm not sure what's silly? The compute bound workload, is now done on the GPU. The IO workload is still done on the CPU, in an inherently single threaded fashion. Even when the multi process computation was done on the CPU, load and store operations were still single threaded. This stands to reason since there is no advantage in splitting 500 concurrent hosts connections into 500* CPU cores connections to hit a central data repository with...
I can't think of any code off the top of my head that calls gethostbyname repeatedly. Maybe a network server of some description which is doing reverse lookups to allow for logging purposes? Although that seems inefficient, I can't think of a real time use case for the host name when you're already in possession of the IP, I can only think of logging / reporting uses cases which would be better served doing the lookup after the fact / offline.
If that's a valid example of what you're suggesting, then would the existing threaded code not be more efficiently implemented asynchronously? There's a finite limit to the number of threads you can create and schedule for these blocking calls, at some point you will have to introduce an async tactic. At that point, why not drop the threading altogether?
You say you would rather build a model on top of threads. Why? Does it make your testing simpler? Does it reduce the time for new starts to get up to speed with your code? Does it reduce the SLOC count? Is it simpler to reason about?
I hope you would agree, in all these cases and many more, threading is at a significant disadvantage. I stand by the assertion that its dead(-ish).
The ish qualifier comes from another case we've not discussed, yet!
I work on a product (network device) which involves all of these activities, and they are all memory latency bound. The overhead of task switching is far too high to recover any benefit from task-switching during memory stalls.
To top it off, the product performs a significant amount of computation, almost none of which fits a SIMT GPU model (i.e. there is a lot of branching).
The only performance solution for our product available from today's hardware is CPU parallelism.
You'd be surprised. Furthermore, I call BS.
So yes I would be surprised. Surprise me?
Calling BS on?
And it's not the "oil and gas industry" or finance, which might have been your expertise and might use GPUs, but are nowhere near being even a large minority of Python use.
It's scientific computing. This is Python's largest niche that needs to parallel compute problems.
Not likely, using Twisted deferreds and a sane cache-wrapper with herd awareness -- you probably want this regardless of long-running cachables.
Of course, Python 3.2 also has a futures [thread or process] builtin, if that's your thing.
10-20 lines of code.
On the other hand, for batch processing, memory can be a big issue (CPU less so).
CCP uses twisted which manages to help you with yielding when doing async IO keeping as much work off the GIL when you are waiting on data and sockets and builds in cooperative multitasking concepts to let you yield to other work, but it's internally not multithreaded or multiprocessing out of the box. You still have scale up worker processes in some cases (usually one per CPU you have) to really make it effective.
Unfortunately, PyPy isn't a viable alternative for most scientific computing problems...
- There wasn't up to date documentation about basic topics, like how to compile it.
- There isn't comprehensive documentation about what RPython is supposed to be.
- In the repository, it's hard to figure out what is RPython and what isn't, what's code for the interpreter, what's code for the stdlib, etc. Specially because modules import from each other in crazy ways.
- It's hard for someone who's not a contributor to peek at the code to figure out why client code isn't running on PyPy, since the only people who understand the architecture are the authors.
- The code itself is pretty opaque and light on comments.
I understand it's a fast moving project and that it had major rewrites so far, but those are the reasons why I say it's not a viable alternative for production.
Do you even know the author? I'm pretty fucking sure he does know about PyPy and Twisted. He is a HN regular, a good tech blogger, and he has TONS of experience with Python for production use.