Hacker News Reader

Comment by keypusher | Hacker News Reader

concurrency in python is kind of a disaster, in my opionion. there are a lot of different options but they all seem to have significant drawbacks, and not just limited to ease of use. i know concurrency is a hard problem, but i wish there was one really good, straightforward solution instead of 3 or 4 different half-baked convoluted solutions (threading, multiprocessing, asyncio, subprocess in stdlib, plus twisted, gevent, pulsar etc as third-party).

View on HN

From what I see, the community keeps changing its mind every few years. We need to pick a multitasking method and really embrace it. If we want to keep the GIL, then we need to go whole hog on some other approach.

A community which changes its mind every few years is a community which does not have effective leadership. Python's leadership is ineffective because 3.x has erased its credibility. You have maximum one opportunity to change a language in mid course. That opportunity has already been used (wasted?) in the case of Python, and while it could have addressed the elephant in the room, multicore, it did not. Instead it indulged the wishes of its powers that be to tinker on the edges to make Python more "serious", in the process adding nothing that anybody needed, while reducing simplicity and accessibility, and, criminal sin, no backward compatibility with no corresponding forward killer feature.

Now, nothing less than a wholesale change of the leadership can right the listing ship.

> Now, nothing less than a wholesale change of the leadership can right the listing ship.

I'd suggest you to go to python-dev and post a concrete (and realistic!) proposal on how to improve whatever you think is broken.

Suggesting to change the leadership isn't really productive.

Leadership change is the fundamental basis of our societies, from the central tenets of democracy, to the structure of our economies where positions of responsibility require competence, at the risk of rapid replacement. It is, on the contrary, very productive to make a case in a public forum (this one) for an ineffective leadership to be changed, if it will help to alter the course of an endeavour which has lost its way, an endeavour, by the way, on which millions of programmers' livelihoods depend.

With due and high respect for the people who work hard on Python, I am happy to contribute, but I am not happy serving the leadership of the project as it currently stands. Python 3.x is a net value-destroying proposition, but the current leadership has spent so much energy exhorting the community to move, without success, that it has no choice now than to stand down.

I suspect you don't really know what you are talking about...

What is the point of this comment? How am I incorrect? My points are:

1) Python 2.7 is still, 6 years later, much more popular than Python 3.x.

2) Python 3.x now has 5 versions, none of which has features which are strong enough to cause migration. Specifically, async is not as easy as the competition, type annotations are an itellectual indulgence if they don't improve performance, and ease of use, a central pillar of the language's attractiveness to newcomers, is being eroded with unnecessary moves towards "seriousness".

3) It was a mistake by the Python authorities to break compatibilty without bringing substantial new features. Breaking compatibility allows revolutionary features. Instead we have breaking of compatibility with only incremental improvement.

4) Because the leadership refuses to acknowledge 1-3, its credibility is now questionable. It is an important related issue that people who point out these problems, as I do, face a strong barrage of criticism, with suspicions of orchestration.

5) With due respect for and gratitude to the inventors and contributors, the time for change has come. For Python to flourish, new leadership is needed.

I am happy to debate my strong opinions, but to suggest out of the blue that I don't know what I'm talking about is not interesting. I have been using Python for a decade. I know it very well. You may know it better, in which case I would like to hear your considered opinions on my points. The context, for avoidance of doubt, is my love of Python, and my strong desire for it to continue to do well.

i personally don't care too much about how python 2 is still more popular than python 3. it's like saying java 1.5 is more popular than java 8. it just doesn't matter to people who use python 3. after working on a python 3 code base for almost 2 years now i wouldn't go back to python 2 if they paid me for it and there's one reason for that: exception handling. it's something that's just broken in python 2. it's a huge change and your 3) is invalid because of that (and unicode, but you probably don't care if you don't consider this a worthwile change.) your 4) thus doesn't follow. re your point 5), python's current leadership should be commended for creating a language so good that people have no reason to switch from. python 3 is even better, so just start using it for new projects.

FWIW nothing stops you from forking the project and being a leader of it. Go ahead, show the better way.

We see people hyping all the time for NodeJS mainly because it can handle millions of concurrently open connections. Now we get a much cleaner (IMHO) way to achieve the same thing in python (of course python is slower) - i.e. see aiohttp as websockets server. I would not call it a disaster.

This is like the cripple making fun of the blind man.

When your competition is NodeJS there is nothing really left to say.

It's great that it works for you for this one particular type of application, but the whole world isn't io bound. Some of us just want regular, plain old, concurrency. As this shows, multiprocessing is still the only, unsane, solution right now. I just want to be able to make a thread, and do stuff in it in parallel to other threads, like I can with most other languages.

"Go write a C extension" you tell me, "use something besides cPython" he says, "just use multiprocessing" I hear. Sure...but ffs, we've had multicore processors for almost TWO DECADES now.

One of my biggest, and apparently unchanging, problems with Python is the desire to keep things simple in the interpreter, to the disadvantage of the language. Sure "implementation for interpreters may vary" blah blah, but you have to target the bottom end in performance, and most widely installed, which is definitely cPython for both points.

I think these are two distinct use cases. NodeJS actually does not (AFAIK) handle one of them. Basically, you have IO bound tasks and CPU bound tasks (and a mix of both which is really nasty business). Python has had CPU-bound task concurrency via multiprocessing and it's been OK. My preference would be to get rid of GIL and improve how threading is actually done, but technically you can serve CPU-bound tasks today with Python 2 and 3. This is (AFAIK) not something that Node does out of the box.

The IO bound tasks in Python are a problem and I wish there was a clean solution. Python does not have a global event loop, so there is not an easy place to hook in coroutines, callbacks, etc. So for a while we were stuck with one of the following:

1. Use threading or multiprocessing. This sucks for more than concurrency of like 2-8.

2. Use eventlet, gevent, or another event loop. The problem here is that you have to buy into it whole hog. No component of yours can be blocking, and that's hard to tell.

3. Write your own event loop. I've done this and find it to be the most understandable and easy to debug approach. This sucks because of the amount of effort it takes for something so fundamental (because networking is tricky).

Some people would be happy if Python got better at solving IO-only bound tasks. I guess that's where this feature comes in. I haven't played with 3.5 yet because I am mostly stuck on 2.7 for reasons. However, looking at it, I feel like there should have been more of a separation between blocking and non-blocking code here. Something alongs the lines of an async function not being able to call a blocking function.

Re: CPU and IO bound tasks: I know of no great framework for this besides threading (not the kind in Python + GIL, but real threading). I usually just side-step this problem by separating tasks that are both IO and CPU bound into smaller tasks that are only CPU or only IO bound. Thankfully, that's generally pretty easy to do.

asyncio provides an event loop in the standard lib now

Try this: http://dask.pydata.org/

And ironically, nodejs is also single threaded with it's own GIL, which somehow no one ever mentions. It's great for async IO bound stuff but is really worse that Python for mutlicore CPU bound work - at least Python has concurrent.futures which makes throwing together a parallel map operation a few lines of code.

This isn't a problem unique to Python. There are a plethora of concurrency solutions in every language because there isn't a one-size-fits-all generic solution for every workload that exists.

It's unique to Python in that most other languages used for more than just scripting have true concurrency via OS preemptive threads with no GIL.

Python is really decades behind in this area and no async hack is going to improve matters. I think it's safe to say that this is an obvious example of how Python is not really a general purpose language (disregarding the fact that it started out as a christmas hack and really has no design behind it). It should be used for scripting and quick prototyping but architecting anything substantial on it is not advisable.

I think this a very unfair comparison.

Everyone who works with Python who understands its limitations has usually found reasonable workarounds for the things you would expect the language to bend to, much like other, similar languages that occupy that niche.

By the same token, we can claim that C++'s horrendous and undecidable syntax and footguns make it a language inadequate for "general purpose" programming; and for many cases you would be right!

But there is no such thing as a language that can do everything well. The other big contender for "general purpose" is the JVM languages, which also suffer from innumerable issues such as slow VM startup time, long VM warmup, enormous RAM usage, lack of very reasonable primitives such as unsigned integers, etc. etc.

Those are the reasons why very large systems either use a hybrid of languages and runtimes for different tasks, or use a monolythic solution and make the adequate compromises.

No one will disagree that the multithreading story would be easier in Python if it chucked the GIL out of the way, but then again, if you're doing that you might as well start a project in Elixir, Julia, Rust, or any number of modern languages that don't suffer from the cruft, but easily will need a decade to catch up in terms of library support and tooling.

> architecting anything substantial on it is not advisable.

If you're in Toronto, Canada in November you should come to my talk at Pycon CA.

Openstack is a rather large Python codebase that powers some rather large public and private clouds from Rackspace to HP and even CERN. I'll be discussing the various technologies and processes that allow us to do that in Python.

It's also one of the more popular platforms for scientific computing and data analysis. There are huge code-bases designed in Python that work quite well and continue to grow in adoption and usefulness. I don't see why anyone would advise against writing a substantial system in Python.

Concurrency is just difficult, with no single solution fully addressing every use case. There are a few mature approaches to python concurrency, their weaknesses lie in the fundamental tradeoffs rather than buggy code. Not that there isn't room for improvement, but its almost impossible to have one solution that solves everything.

Then why do we have an official concurrency module? If there is "no single solution fully addressing every use case", why do we have one official solution? Which the main post criticises? If you are correct that there is no single solution, then surely there should be no single solution put forward as official?

Yet I believe your premise is incorrect in the first place. Why has Golang, by most commentators' comments, "got concurrency right". Is that true? Tell me. If it is, why can't Python get concurrency right also? If it's not true, why is there an official solution, being criticised before our eyes?

On this case, I will quote you: "I suspect you don't really know what you are talking about...".

Reply on news.ycombinator.com