Hacker News Reader

Comment by gmueckl | Hacker News Reader

Hm, this solution seems very cumbersome, inelegant and not like python's "batteries included" approach at all. This means that python will have native threads that behhave as expected minus true parallel execution, so you shouldn't use those, even though the interface is fairly simple. Instead, you should learn to use this weird contraption that is neither multiprocessing nor intuitive multithreading and comes with a cumbersome interface.

I get that the GIL is a very hard problem to solve, but this solution is so inelegant in my eyes that python would be better off without it. I'd feel better if this was a hidden implementation detail that coukd be improved transparently. Just my two cents.

View on HN

>This means that python will have native threads that behhave as expected minus true parallel execution, so you shouldn't use those, even though the interface is fairly simple.

Python already has exactly that, and has had that for ages.

>Instead, you should learn to use this weird contraption that is neither multiprocessing nor intuitive multithreading and comes with a cumbersome interface.

It also comes with performance improvements over multiprocess, so there's that.

Besides the "cumbersome interface" is irrelevant, as it would be easy to wrap and forget about it, the same way nobody really uses urllib directly.

> Python already has exactly that, and has had that for ages.

And the point is that this should be fixed, instead of adding yet another clunky way of doing the same thing.

> Besides the "cumbersome interface" is irrelevant, as it would be easy to wrap and forget about it, the same way nobody really uses urllib directly.

Then why bother having it, if the only reason for it to exist is so that it can be wrapped? That's just stacking layers of turd.

Almost nobody writes assembly nowadays, but I don’t see anyone wanting it gone.

This is assembly. Don’t want to wrap it yourself, fine, don’t complain that it exists.

You could implement requests without urllib. You cannot implement C without assembly. That's the distinction between useful layers of abstraction and pointless ones. If a better interface to subinterpreters is bound to be developed, then that should've been the stdlib interface in the first place.

>You could implement requests without urllib.

Yes, but you couldn't do more fine-grained than requests things, unless you did them from excruciatingly lower level (e.g. sockets).

>And the point is that this should be fixed, instead of adding yet another clunky way of doing the same thing.

Well, unless we have a (1) capable (2) volunteer the point is moot.

Those who actually work on Python development accessed both scenarios and found adding the "yet another clunky way" if not optimal, a more feasible, easier and better use of their limited time.

>Then why bother having it, if the only reason for it to exist is so that it can be wrapped?

Because lower level primitives can be used in many useful ways than some constraining higher level construct, but the latter is still nice to have.

I have the same opinion about asyncio, its such a bad API that its almost impossible to use correctly. But still, probably better than nothing.

Use curio or trio instead :)

async/await are not only for asyncio.

Coming from the JS world, asyncio feels fine to me. I've noticed a lot of Python programmers who have a strong background in C and Python struggle with asyncio. But in their case, I think it's more them disliking the async programming paradigm rather than the specific asyncio API.

personally I have JS experience (browser/node.js), as well as asio in C++/Boost, EventMachine in ruby and Twisted/Tornado in python. I strongly disagree that people who dislike asyncio only do so because they don't like the event loop. That just sounds like a typical quick dismissal of criticism. No, its a objectively awful convoluted API.

But if you think you can write correct asyncio applications of any significant complexity, that's great. Doesn't change the fact that a lot of highly experienced developers struggle to do so.

> But if you think you can write correct asyncio applications of any significant complexity, that's great.

To be fair, I've never written an asyncio application of any significant complexity (at least compared to the async JS application I've written). But nothing jumps out at me as horrific with asyncio. The worst thing is the split between futures and coroutines, but once you realize that there's no real need to use futures any more with asyncio (although you can integrate futures-based libraries if needed), you're good to go.

But fair criticism, I've only written small asyncio applications. The only sizable async-Python application I've made was made using Tornado and ZeroMQ async APIs(although even that eventually ran on the asyncio event loop and used an asyncio-based library).

I completely disagree - Python threads are basically "green threads", so they have their place but aren't related to parallelisation. But true multiprocessing is ugly when you have hundreds of cores, which is where CPUs are going. There is no standard UI convention on most OSes to group those processes per app, in terms of signals or stats or whatever.

So besides the unproven possibility of removing the GIL, subinterpreters are the best way forward, better than threads or the multiprocessing package.

It's not that Python shouldn't have subinterpreters, but that they should only be added when they have a better API - at least as good multiprocessing.

I don’t think they’re green threads - they’re full OS threads. They just have the GIL.

> Python threads are basically "green threads"

That's not accurate.

> they have their place but aren't related to parallelisation

You can parallelize all sorts of things with Python threads--just not some things you'd expect to be able to parallelize, due to the GIL. Waiting on or buffering I/O, calling out to compiled code, doing cryptographic operations--all of those can be parallelized, as (in many cases) they entail releasing the GIL.

> But true multiprocessing is ugly when you have hundreds of cores

Why?

> There is no standard UI convention on most OSes to group those processes per app, in terms of signals or stats or whatever.

I have no idea what this means. What do UIs have to do with process groups? Do you know how many processes your Chrome instance is running on the operating system? There are very solid conventions regarding process management, at least on Unix-ish systems: process groups and parent-child relationships are well established and well understood, as is their relationship with signals and signal handling.

I think he's saying that they allow you to do concurrency, but not parallelism. Those are two different things.

They do allow you to do parallelism, though. There are just some things they can't parallelize because of the GIL.

Green threads also allow for parallelism, if they're scheduled onto more than one executor.

That's true, behind the scenes they can be running on multiple CPUs, which is the definition of parallelism.

I think threads are a bad abstraction for doing parallelism personally, though. Programs designed to run in parallel should be deterministic, unless they need concurrency for some other reason. I think trying to shoehorn parallel programming into Python's threads isn't necessarily the best approach.

As far as I'm concerned, if I'm using threads and they happen to get scheduled on to multiple cores, then that's a nice optimization, but isn't necessary for what I use threads for.

If that’s what he’s saying, he’s wrong. Python threads are based on native threads. They are parallel executed, they just don’t do so very well because of the GIL.

Reply on news.ycombinator.com