Hacker News Reader

Comment by LtWorf | Hacker News Reader

I mean… if you have unlimited resources and unlimited speed, going multiprocess makes complete sense.

But if you don’t, you don’t use Python anyway

Maybe you have finite time to do the implementation and your task is IO bound? (which is where using async makes sense, by the way)

Lets make a specific example. You have an APIs to services with data that you want to synchronize to a cache in a local database. The providers allows a maximum of 1-3 connections for each api.

You can write this in async with a pool of connections to each provider, or you can have a pool of worker processes that each have one connection to each provider.

Which will be easier to debug, and which design is easier for the next developer to read, understand and modify? Performance wise the wast majority for both designs will be on waiting at the initial connections to the apis and time spent fetching objects.

Now let's change the example but bump the limit to 3000 connections.

And if the there were 3000, 30000 or 300000 providers the problem would be different. The example I gave was one where I have considered async but went with multiprocessing, which is also why I asked the parent commenter above if they found any direct benefit to readability or debugging when going async.

It would be interesting to hear what kind of domain are you working in where you have 3000 simulations connections to different service providers.

Reply on news.ycombinator.com