Hacker News Reader

Comment by arp242 | Hacker News Reader

I was using gevent in Python about 10 years ago, and from memory, it's roughly similar to goroutines. It's not exactly the same of course, but just like goroutines it's pretty easy to just spawn a few jobs off, wait for them to finish, and get the results.

It's not in the standard library, and there are probably other options too now (not a heavy Python user any more), but Python has had easy parallelism for at least a decade (probably longer).

View on HN

Gevent is like goroutines with GOMAXPROCS=1. Which is to say not nearly as useful. It gives you concurrency without parrallelism, because Python never did shake the GIL. Which goes to show some technical debt will haunt you forever.

To nuance your comment, you can still get some form of parallelism, "just not" thread parallelism in Python. You can still spawn multiple handler process, or have threaded code in a C extension.

It’s arguable whether either of those are ‘in python’.

The multiprocessing module in the standard library is absolutely a Python-native way to do parallelism:

    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

This runs f(1), f(2), and f(3) in parallel, using a pool of five processes.

https://docs.python.org/3.6/library/multiprocessing.html

Whoa! It never occurred to me that `Pool` could be used with context manager. I've always typed out `.close()` manually like a sucker.

https://docs.python.org/2/library/multiprocessing.html

Python standard library since 2.6. Pretty much the definition of "in python".

Yes, you can use multiple processes to get the parallelism. But that's quite limiting compared with goroutines. Passing data back and forth is hard, and you can petty much forget about shared data structures. Memory usage is also much higher.

The GIL is not a total ban on thread parallelism. It's a significant obstacle, but not a complete stop.

Besides, Go has its own set of problems with parallelism. None of those are best in class.

This comment makes it sound like Go and Python are pretty much in the same class because neither is perfect. I invite anyone who thinks this way to write considerable (shared-memory) parallel code in both languages and see which they prefer.

As for Go's "set of problems with parallelism", they're pretty much just that sharing memory is hard to do correctly without giving up performance. No languages do this well; Rust and Haskell make it appear easier by making single-threaded code more difficult to write--requiring you to adhere to invariants like functional purity or borrowing. If you're writing Python, you very likely have values that are incompatible with these invariants (you want to onboard new developers quickly and you want your developers to write code quickly and you're willing to trade off on correctness to do so).

Go is absolutely best-in-class if you have typical Python values.

> Go is absolutely best-in-class if you have typical Python values.

"Best in class" is not a relative term. Neither Go nor Python are appropriate choices for highly parallel intercomunicating code. Yes, Python is more limited than Go here, but hardly makes a difference when you avoid it.

Haskell and Rust do make it easier, by forcing developers to organize their code in a completely different way. Erlang does the same, with a different kind of organization. None of those languages are more difficult to program in, but yes, they are hard to learn.

> Neither Go nor Python are appropriate choices for highly parallel intercomunicating code.

thats an extraordinary claim which needs evidence.

Extraordinary?

Name any single Go feature aimed at helping parallel computing.

Why bother writing shared-memory parallel code if it makes your life so hard? Most of the time you're i/o bound, or network bound, or storage bound. Being compute bound is exceptionally rare these days.

I do it when I have to do it. Most things are I/O bound, but sometimes things are compute bound. And since Python is about two orders of magnitude slower than Go (and Python also makes it much harder to optimize straight-line execution because it lacks semantics for expressing memory layout), you tend to need parallelism more often than you would in a fast language. Sometimes you can leverage pandas or write a small chunk in C, but very often those options aren’t available, and naively throwing c at the problem can make your performance worse.

> Gevent is like goroutines with GOMAXPROCS=1.

On its own, yes. For webapps, you can easily combine it with a multi-process WSGI server (like gunicorn or similar).

Running four processes with GOMAXPROCS=1 is strictly worse than 1 process with GOMAXPROCS=4. The difference is coarse-vs-fine grained parallelism. Notably, a WSGI+gevent system doesn't allow you to do parallelism within a request, not to mention configuring these WSGI implementations (especially for production) is a bunch of extra headache for which there is no analogy in Go.

Funny you mention the GIL - this is being done as part of PEP 554 which was slated to release in python 3.9 (in alpha).

And it's showing great promise ...while it could be delayed.

https://www.mail-archive.com/python-dev@python.org/msg108063...

2021 is probable going to be the "Gone Gil" moment !

Why don't they use/fork the GoLang runtime?

There are a few reasons: 1. This would definitely break the CPython API, which is not an option for mainstream Python. 2. The Golang runtime isn't really well-understood as a backend for languages that aren't Golang, although I acknowledge that there's no reason in principle why you couldn't compile $LANGUAGE to Golang.

I still use and greatly prefer gevent to this day. They are indeed quite similar to goroutines. Asyncio is a different model, and more irritating for me to work with. (I'm sure this isn't the case for everyone. I'm just more productive with gevent, personally.)

Performance is pretty close for both. I was disappointed to not see Python get an official green thread implementation. The counter-argument I commonly see cited is https://glyph.twistedmatrix.com/2014/02/unyielding.html. I personally don't find it to be a very convincing argument.

What's missing in asyncio when compared to gevent ?

The coroutine and queue model is the same right ?

Cool thing in 3.8:

Running python -m asyncio launches a natively async REPL.

They're just completely different models. gevent is green threads, asyncio is explicit coroutines. In gevent, you don't use an "async" or "await" syntax. You just spawn greenlets (green threads), which run in the background, and if you want, you can have them block until they return a result. And they're efficient, so you can spawn thousands without a problem.

I always considered "green threads" and "coroutines" to be the same thing? Are they not?

They are both built on the same technology in the CPython runtime.

The difference is that with coroutines, yielding is explicit. Tight-looping will hog the CPU, and a blocking call will block other coroutines too. Typically "green threads" are semantically just threads but cheaper. They're scheduled independently, so there's no risk of them hogging the CPU, and you can use synchronous apis. The one downside is that you need explicit synchronization between them, whereas with coroutines you can mutate shared data structures and not worry about race conditions as long as you don't yield in between.

If you have a significant python code base that is not async, then all of that need to be ported to support async model where as with gevent I can do monkey patching and move to concurrency model. If I am starting a fresh project with python and need concurrency, yes "async" is a better choice, but if you already have some code base then moving to async is a fair amount of work.

With asyncio, your whole app falls over if you accidentally call a library function that makes a sync API call under the covers. gevent (as I understand it; haven't actually used it) will patch all sync APIs and make them async. Also, if you do `aiohttp.get("www.example.com/foo.json").json()", you get a TypeError because coroutine has no method '.json()' (you forgot `await`) unless you're using Mypy.

Yep, that about sums it up. gevent can't monkeypatch synchronous code that's implemented in non-Python native modules, but I think pretty much all native Python libraries struggle with those sorts of things, and asyncio of course also can't deal with it.

The vast majority of the time, gevent's monkeypatching works without any issues. With asyncio, you basically have to rewrite everything from the ground up to always use the new async APIs, and you can't interact with libraries that do sync I/O.

Reply on news.ycombinator.com