Hacker News Reader

Comment by Spivak | Hacker News Reader

What is Python missing in your eyes? Cooperative multitasking is a relatively recent thing to be in vogue again, and python has had good support for multithreading and multiprocessing since forever.

View on HN

Without in-process multi-threading with OS threads, writing performant parallel code is nearly impossible. Most of the time you have to drop down to C/C++, disable the GIL and implement parallelism there.

Every time someone brings up the GIL and performance, the answer is pretty much always "use a different language for the places you need especially high performance". All this sturm and drang about Python not being performant enough is missing the point. It's not supposed to be the answer to special cases where maximum performance is required! It's a general-purpose language that is built around community and easy readability and elimination of boilerplate, and be good-enough for a vast majority of usecases. Just like any language, there are usecases where it is not appropriate. Clearly, it has succeeded given its significant reach and longevity.

Most high-level languages implement their performance critical functions at a lower level. However, languages that rely on a GIL (Python and Ruby) or isolated memory spaces (V8) have an additional hurdle that if you want a model of concurrency with many threads acting simultaneously on a single shared memory space you have to do additional work.

For Python you either have to have your library pause execution of Python bytecode and do multithreaded things on your memory space, allocate an isolated memory arena for your multithreaded work, copy data in and out of it, and then spawn non-Python threads (see PEP 554 for an IRL example of this idea with the Python interpreter itself), or copy data to another process which can do the multithreaded things.

With PEP 554 (right now using the C API) I personally have no issue with multithreaded Python since the overhead of copying memory between interpreters is fast enough for my work but it is overhead.

There is nothing about the Python language spec (as gleaned from looking at how CPython works) that forces it to be slow or not handle parallelism. In fact the addition of the massive amounts of library code and language changes to support async show that Python isn't even immune to preventing added complexity.

Also dask works pretty good all the way upto 200-250 CPU cores. It's pretty straightforward to build your code to use dask rather than numpy or pandas

Reply on news.ycombinator.com