Hacker News Reader

Comment by stickfigure | Hacker News Reader

I don't really understand the problem with the GIL. Sure, people doing exotic high performance stuff probably hate it, but Python probably isn't their go-to language anyways.

For boring old web development and business processing, the GIL seems to be a very efficient way of utilizing a CPU+caches without resorting to the horrible event-driven programming model of Node. Want twice the throughput? Run twice the processes. I kinda wish Java would bring back the option of green threads, it was a perfect programming model for web development.

View on HN

The problem is with CPU intensive tasks. Being as expressive as Python is, it would be nice if CPU intensive algorithms could be written in it, without blocking all other Python threads while the algorithm runs. For example.

>it would be nice if CPU intensive algorithms could be written in it

It would be nicer if instead of trying to turn python into something it is not (a language for writing CPU intensive tasks), it was made better at what it is (glue code).

It would be especially nice if there were tighter, more seamless integration between python and rust so that hot paths in python can easily be migrated to rust.

No to threading, but yes to tighter integration with rust.

Ah yes, let's not focus on something python is not. Let's appeal to the majority use case.

What minor planet are people here from?

Python is far more than glue code.

I'm not a Rust guy, so take this with a grain of salt, but I was under the impression that Rust did a great job with C interoperability, which is all you need for writing Python extensions. Some hasty googling brings up https://github.com/PyO3/setuptools-rust, which appears to be exactly the thing you're asking for. In other words, it's not heavily advertised, but it appears to be entirely feasible to build Python extensions in Rust, and not spectacularly complicated either.

>Some hasty googling brings up https://github.com/PyO3/setuptools-rust, which appears to be exactly the thing you're asking for.

Yeah, I've tried it and it didn't work.

It's certainly feasible it's just not very well developed.

The language does do great at the core of it, but there's a lot of stuff that makes it easier that's not the language's job.

compare https://doc.rust-lang.org/1.2.0/book/rust-inside-other-langu... to http://usehelix.com/ for example; the former is only using what the language knows, the latter is a nice library built on top.

There's been some effort for nicer Python integration, but I haven't seen anything super super slick yet.

If it's really CPU intensive, you can always fork.

The problem is the overhead of memory copy.

Linux is somewhat smart about it with its copy on write semantics (a double edged sword really). However, the moment you refer to an existing Python object the underlying data-structure gets mutated because of reference counting and boom now you have to copy the whole thing.

This can be worked around with creative stealing and by keeping a sharp lookout for references but it gets really painful and annoying.

Even if that were resolved its still not as flexible as one would want. One cant really go about spawning and terminating process too frequently. This forces one to move larger and larger parts of the logic into C.

> Python probably isn't their go-to language anyways.

That is the real problem. Any sufficiently popular language will inevitably be used in scenarios it is not suited for. Python has now reached that threshold, so all sorts of communities are trying to stretch it and pull it every other way. Sometimes this makes the language better, sometimes it's just a source of complication and frustration.

There is also the issue that some quarters feel python's role is being eroded by newer stuff like golang and nodejs, which were built with high-performance multithreading in mind from the start. I personally think this is not much of a concern (if you are willing to trade python for golang or nodejs, you were not really using python because of its expressiveness and conciseness in the first place), but clearly a lot of people do.

What other tradeoff targets would be the considered as sharing python's expressiveness and conciseness? One of the python variants like stackless? C?

I'd be very curious as to the answer here. Some times you do like the pretty code from something like Python, but you get backed into a performance corner, at least I could see that happening.

I don't even think it's about pretty code - I think it's more related to what one sibling of the post you replied to mentioned, which is that there are tons of communities (among them machine learning, but including tons of stats-heavy academic programming) which would REALLY like to do some extremely heavy work in Python, not because it's necessarily the best, but because it's what they know. Academics in particular (not CS academics, but stats and experimental sciences people) can be forgiven for wanting to stick with Python - they aren't programmers, they don't want to waste time learning programming language skills that are largely irrelevant to their job. All they want is to pick up an easy language and be able to do their work in it, but once their work gets heavy, the language starts to fight them. And that's a really popular use case for Python, I think.

Many machine learning people use Python, and they need all the CPU power they can get.

Python added the "horrible event-driven programming model of Node" in Python 3.6. The syntax is said to be better, though.

The python libraries ML people use (scikit-learn, tensorflow, numpy) all do the heavy work in C though

Yes, but the light work, run on a single Python thread, will in some cases take a significant part of execution time.

If you have 16 CPUs in your system, seeing your load at 6% for the first 10% an last 10% of running time, and at 100% in-between surely will make people think there must be easy wins in that Python code. More so since many users will have effectively _zero_ knowledge of C.

Many machine learning people use Python, and they need all the CPU power they can get.

The compute code for ML is emphatically not written in Python – Python is essentially a lightweight API wrapper around it.

That's bit of a circular argument.

Agreed that compute code for ML is not written in Python, but if GIL wasn't a problem there would be one less reason to drop down to C. Many users would appreciate that. Cython helps with the C part but to a degree.

You are forgetting python is (depending on the benchmark) still 100-400x slower than a language compiled to native machine code. Removing the GIL might slightly help in some cases but anything CPU bound requiring high performance will still need to drop down to something faster.

That's a CPython problem. PyPy is much faster at number-crunching.

Its a perception problem. I think the simple perception of the GIL as making multithreading untennable in Python has given Go a big boost and has hurt Python considerably.

>has hurt Python considerably

Has it? Why the assumption that Go is a direct competitor?

My impression is that Go excels where Python does not, and vice-versa. I use both languages a lot for my day-job, and when using Go, I very much miss having an interpreter and a fully-interactive debugger (Delve is excellent considering the limitations imposed by Go, but much less flexible than pdb).

And let's not even talk about metaprogramming...

Anecdotal, but in 2010 the creators of Go noted in a blog post [1]:

>We had anticipated interest from C, C++, and Java programmers, but the flurry of interest from users of dynamically-typed languages like Python and JavaScript was unexpected.

[1]: https://blog.golang.org/go-one-year-ago-today

That tells us people who use Python are using Go. It does not tell us that Python is suffering as a result of this.

In my experience, these people are using both languages for the reasons mentioned above.

Reply on news.ycombinator.com