The GIL allows only one OS thread
to execute Python bytecode at any
given time
So on top of being 6x slower than PHP and Javascript, Python does not benefit from more than one processor core?
So in a web server environment with - say - 4 cores, it will be 24 times slower?
However, most of the time you’re doing CPU-heavy tasks in Python you’re likely using libraries that are ultimately implemented in another language for performance. The GIL can be (and is) released in those cases. Not much of a problem there.
It’s also released when you’re waiting for I/O. Not much of a problem there either.
Fear of the GIL have never stopped me from using threading in Python, and it has never been much of a problem. In the rare occasions I’ve noticed, I was doing something wrong, not using the appropriate library, or both. Or it might be the case that Python was just a poor choice for the problem, no matter how flexible it is.
I also wonder what would happen if CPython got rid of the GIL. Python has certain characteristics that make it hard to have a parallel interpreter, and people have tried (and failed) to remove te GIL before.
Don’t fear the GIL.
Parallelism strategies in Python usually revolve around using the multprocessing module, a c extension (numpy will release the gil, and hence all the science stack) or using a 3rd party tools (dask is pretty popular, and allow easy parallelism: https://dask.org/).
I have the feeling what the TFA really means is something different. But I cannot really figure out what.
Naturally you can have multiple OS processes of Python that run in parallel. This is how Python web services usually work. Then, of course, you need to share memory explicitly via some mechanism like Redis or SysV shared memory, so it leads to different design considerations.
You can also have a single Python process running multiple threads in parallel, as long as all but one of those threads are currently running C code (numpy, system calls, ...) But if your program spends more than a tiny fraction of time in interpreted Python code, the GIL will slow you done if you try to use all cores.
You can have multiple processes running Python at the same time -- and indeed, many Python programs work around the GIL by forking sub-processes. This tends to introduce significant extra complexity into the program (and extra memory usage, as the easiest solution is typically to copy data to all processes).
It does not. It states that two python threads can’t execute bytecode simultanously, the implication is that that’s within a single process (which is the case) otherwise talking about threads makes no sense.
> I have the feeling what the TFA really means is something different. But I cannot really figure out what.
TFA means exactly what it says. Two threads can’t execute python bytecode in parallel, because there’s a big lock around the main interpreter loop (that is in essence what the GIL is). That is quite obviously within the confines of a single Python interpreter = runtime = process.
This is a really strange perspective, given how many shops use python for production these days.
> Ruby is a bit better, but these days there's no reason not to use Go or Rust.
Unless you're specifically looking to access for example machine-learning resources, that are not nearly as mature in those other languages.
No it wasn't. It was supposed to be a language for automating sysadmin scripts, or for embedding in a program to run user-scripts.
PHP does not have a GIL, thus PHP does not suffer from any slowdown when using threads, proper use of threading improves PHP performance without the need to escape to a C library. Internally PHP uses something called Thread-Safe Resource Manager (TSRM) when built as thread safe. TSRM architecture was improved with PHP 7 and to my understanding performance as well.
However PHP does not ship with any API to handle threads, this is because normally you, the web programmer, don't do any process/thread handling in a web setup, concurrency is handled by the webserver for each request. It does exist third party extensions that enables threading API, like in a cron job.
Good luck doing any deep learning in Go or Rust :)
Different Python processes can certainly use different CPU cores - the sentence is definitely misleading.
(As others have mentioned, it also only applies to interpreting Python byte code itself, not OS calls or C extensions modules called from that bytecode.)
For example, I've seen people write entire programs in C and, after taking several weeks, realize their code was nowhere near as fast as the Python version because the code was bottlenecked on dictionary operations and Python's stdlib version was far more tuned than the C implementation they'd written (repeat for image handling where they wrote their own decoder, etc.). This is not to say that a C programmer shouldn't be able to beat Python — only that performance is not a boolean trait and it's important to know where the actual work is happening.
The important thing to know is that the GIL can be and is commonly released by C extensions. If your programs tend to be I/O bound (e.g. many web applications) or are using the many extensions which release the GIL as soon as they start working, your performance characteristics can be completely different from someone else's program which is CPU-bound and has different thread interactions.
This is why you can find some people who think the GIL is a huge problem and others who've only rarely encountered it. These days, I would typically move really CPU-bound code into Rust but I would also note that in the web-ish space I've worked in the accuracy rate of someone blaming the GIL for a performance issue has been really low[1] compared to the number of times it was something specific to their algorithm.
1. I'm reminded of the time someone went on a rant about how Python was too slow to use for a multithreaded program working with with gzipped-JSON files — 3 months and an order of magnitude more lines of Go later they had less than a 10% win because Python's zlib extension drops the GIL when the C code starts running and was about the same speed as Go's gzip implementation, both running much faster than storage. The actual problem was that the Python code was originally opening and closing file handles on a network filesystem for every operation instead of caching them.
Also what's your source for the claim that Python is 6x slower than JS or PHP?
Is this as good as a high-performance concurrency platform (like, say, the JVM has become)? Will it lead to absolutely optimal utilization of all cores? No. But the result is not simply "24 times slower", and all cores will be in use.
Performance numberes like "6x slower" are almost always contextual when it comes to the real world, it's not like every single program written in python takes 6x longer/6x as much CPU as the nearest-equivalent program in PHP, that's not how it works, it will depend on what the program is doing and how it's deployed.