How to Make Python Wait
blog.miguelgrinberg.com
blog.miguelgrinberg.com
Manually dealing with threads and processes is useful if you want to build a framework or a very complex workflow. But chances are you just want to run stuff concurrently, in the background.
In that case (which is most people case), you really want to use one of the stdlib pools: it takes care of sync, serialization, communication with queues, worker life cycle, task distribution, etc. for you.
Plus in such case, waiting for results is super easy:
import time
import random
from concurrent.futures import ThreadPoolExecutor, as_completed
# the work to distribute
def hello():
seconds = random.randint(0, 5)
print(f'Hi {seconds}s')
time.sleep(seconds)
print(f'Bye {seconds}s')
return seconds
# max concurrency is 2
executor = ThreadPoolExecutor(max_workers=2)
# submit the work
a = executor.submit(hello)
b = executor.submit(hello)
# and here we wait for results
for future in as_completed((a, b)):
print(future.result())
import time
import random
from concurrent.futures import ProcessPoolExecutor, as_completed
def hello():
seconds = random.randint(0, 5)
print(f'Hi {seconds}s')
time.sleep(seconds)
print(f'Bye {seconds}s')
return seconds
# Don't forget this for processes, or you'll get in trouble
if __name__ == "__main__":
executor = ProcessPoolExecutor(max_workers=2)
a = executor.submit(hello)
b = executor.submit(hello)
for future in as_completed((a, b)):
print(future.result())
What about this?
Or you use asyncio for network based stuff.
Only if you really need to. For 99% of network needs, using pool executors is simpler and easier than asyncio and it's one less only-sort-of-useful thing to have to learn.
Plus, I'm working with andrew sveltov on a new API for aiohttp, so that you can do:
def main():
response = await aiohttp.get(url)
asyncio.run(main())
The biggest problem is that I have yet seen a tutorial that explains properly how to use asyncio.
They all talk about the loop, and future, etc..
First, they all should tell you that asyncio should be used only with Python 3.7+. Don't even bother before. Not that it's not possible, but I've done it, and it's not worth the trouble.
Then, all tutorials should mention wait() or gather(), which are the most important functions of the whole framework. It kills me to never see those explained.
With just that knowledge, you can script happily at least as easily as with the pools I just had show case.
Now, I really hope that we are going to get trio's nurseries imported in the stdlib. yury selivanov is working on it from uvloop, so I got good hopes.
I did a proof of concept as an asyncio lib and it works decently, but having a standard would be much, much better.
from concurrent.futures import ProcessPoolExecutor
import string
def hello() -> int:
seconds = random.randint(0, 5)
print(f'Hi {seconds}s')
time.sleep(seconds)
print(f'Bye {seconds}s')
return seconds
# Don't forget this for processes, or you'll get in trouble
if __name__ == "__main__":
inputs = list(string.printable)
results = []
# You can sub out ProcessPool with ThreadPool.
with ProcessPoolExecutor() as executor:
results += executor.map(hello, inputs)
[print(s) for s in results]But be careful, map() and submit() + as_completed() don't have the same effect.
The first one will give you the result in the insertion order, while the later give you the results in the order they are completed.
from concurrent.futures import ProcessPoolExecutor
import string
import random
import time
def hello(output) -> int:
seconds = random.randint(0, 5)
print(f'Hi {output} {seconds}s')
time.sleep(seconds)
print(f'Bye {seconds}s')
return seconds
# Don't forget this for processes, or you'll get in trouble
if __name__ == "__main__":
inputs = list(string.printable)
results = []
# You can sub out ProcessPool with ThreadPool.
with ProcessPoolExecutor() as executor:
results += executor.map(hello, inputs)
[print(s) for s in results]
If you wouldn't mind going into a little more detail about what you're doing I'd really appreciate it!
What do you want to know ?
I often remember useful standard libraries that I hear about by name, after using them once, but this is a long one.
Alternatively - is this an example in the python readthedocs yet? I try to fall back to that when possible, so my code stays simple for others.
What is a good solution that the majority of us simple humans with limited time and resources should pursue? I want to write software that competes with similar solutions written in Go. Does that exist? Is there one solution that it's safe to adopt and generally recommended by the community? I used to think that asyncio was the solution to that but I was unable to fully understand its API. Please help!
futures, uvloop, tornado, twisted, coros, gevent, etc are kinda just all related to, or do similar to asyncio
threading is kinda not as useful as you might like in python because of the GIL (simplistically, assume that python can only do 1 thing at once regardless of having multiple threads available or not until you know why that’s not always the case)
You can tell someone a brief answer to memorize, but to teach them anything takes a little more explaining.
For example, Question #1 is often "how do I do this in some particular way", but 2 and 3, "why" and "when" should I do it that particular way -- they take more explaining than #1 -- and through that explanation, you start to get good at your craft.