Although this is nice, the problems with the GIL are often blown out of proportion: people stating that you couldn't do efficient (compute-bounded) multi-processing, which was never the case as the `multiprocessing` module works just fine.
Once the Python ecosystem supports either subinterpreters or nogil, we'll happily migrate to those and get rid of our hacky interprocess code.
Subinterpreters with independent GILs, released with 3.12, theoretically solve our problems but practically are not yet usable, as none of Cython/pybind11/nanobind support them yet. In comparison, nogil feels like it'll be easier to support.
[1] https://docs.ray.io/en/latest/ray-core/walkthrough.html#call...
[2] https://docs.python.org/3/library/multiprocessing.html#manag...
Sharing arrays of numbers is supported in multiprocessing as well: https://docs.python.org/3/library/multiprocessing.html#shari...
Unfortunately, the closest thing we have to that is Julia, which fails to meet none of the requirements. Alas.
Julia’s threading API is really nice. One deficiency is that it can be tricky to maintain type stability across tasks / fetches.
I was intrigued by Julia a while ago, but didn't have time to properly learn it.
So just out of curiosity: what's the issues with jit and Julia ?
The famous "Time To First Plot" problem was about taking several minutes to do something like `using Plots; Plots.plot(sin)`.
But to be fair recent Julia releases improved a lot of it, the code above in Julia 1.10 takes 1.5s on my 3-year old laptop
I think Python is a terrible language that exemplifies the maxim "worse is better".
"My second [surprise] came a couple of hours into the project, when I noticed (allowing for pauses needed to look up new features in Programming Python) I was generating working code nearly as fast as I could type.
When you're writing working code nearly as fast as you can type and your misstep rate is near zero, it generally means you've achieved mastery of the language. But that didn't make sense, because it was still day one and I was regularly pausing to look up new language and library features!"
Source: https://www.linuxjournal.com/article/3882
It doesn't go for large code bases, but if you need quick results using existing well tested libraries, like in machine learning and data science, I think those statements are still valid.
Obviously not when you're multiprocessing, that is going to bite you in any language.
apart from go (maybe java) those are all "scary" languages that require a bunch of engineering to get to the point that you can prototype.
even then you can normally pybind the bits that are compute bound.
If Microsoft had been better back in the say, then c# should have been the goto language of choice. It has the best tradeoff of speed/handholding/rapid prototyping. Its also statically typed, unless you tell it to not be.
gets you 90% of the potential performance of a full multithreaded producer/consumer setup in C++. C++ isn't as scary as it used to be.
Of course, getting threads to be actually useful for concurrency (GIL removed) adds another very useful tool to the performance toolkit, so that is great.
You can do whatever you want in the workers, I parse JSONs and write to sqlite files.
Not if you use Windows, then it's a mess. I have a suspicion that people who say that the multiprocessing works just fine never had to seriously use Python on Windows.
* All Python webservers that somewhat support multiprocessing on Windows disable the IOCP asyncio event loop when using more than one process (because it breaks in random ways), so you're left with the slower select() event loop which doesn't support more than 512 connections.
1. Because global objects are refcounted, CoW effectively isn't a thing on Linux. They did add a way to avoid this [0], but you have to manually call it once your main imports are done.
2. On Mac, turns out a lot of the system libs aren't actually fork-safe [1]. Since these get imported inadvertently all the time, Python on Mac actually uses `spawn` [2] -- so it's roughly as slow as on Windows.
I haven't worked in Python in a couple years, but handling concurrency while supporting the major OSes was a goddamn mess and a half.
[0]: https://docs.python.org/3.12/library/gc.html#gc.freeze
[1]: https://bugs.python.org/issue33725
[2]: https://docs.python.org/3.12/library/multiprocessing.html#co...
I see this mentioned from time to time, but intuitively you'd think this wouldn't pose a big slowdown since the system builtin objects would have been allocated at the same time (startup) and densely located on smaller nr of pages. I guess if you have a lot of global state in your app it could be more significant.
Would also be interesting to see a benchmark using hugepages, you'd think this could solve remaining perf problems if they were due to large number of independent CoW page faults.
Something that tripped me up when I last did `multiprocessing` was that communication between the processes requires marshaling all the data into a binary format to be unmarshaled on the other side; if you're dealing with 100s of MB of data or more, that can be quite some significant expense.