Hacker News Reader

Comment by nbsande | Hacker News Reader

run_in_executor is pretty powerful for running sync code in async code, but my use case was more making async code utlize the cpu better. I think, just using run_in_executor would add a lot of complication and changes to how you use async await. But great point none the less!

View on HN

I've had plenty of success using run_in_executor in production. You basically just use the decorator, and it mostly just works.

Which decorator?

I went back to look at some of the old code as my memory was hazy. It was tornado's implementation of the run_on_executor method that we used, which is used as a decorator.

https://www.tornadoweb.org/en/stable/concurrent.html#tornado...

> but my use case was more making async code utlize the cpu better

But run_in_executor achieves that as well! If you use no-GIL Python and a thread pool (or GIL Python with a Process pool), you will utilize more CPU cores.

run_in_executor that can be spelled as asyncio.to_thread() won't help utilizing cpu for pure Python code. It may help with blocking I/O, or cpu-heavy C extensions that release GIL. Otherwise, you have to utilize different processes to take full advantage of cpus (there are also subinterpreters (same process, separate GILs) but that exotic).

>run_in_executor that can be spelled as asyncio.to_thread() won't help utilizing cpu for pure Python code

That isn't true. If you use a ProcessPoolExecutor as the target instead of the default executor, you will use multiple processes in pure Python code.

the default executor must be ThreadPoolExecutor. But you are right you can pass ProcessPoolExecutor too.

https://docs.python.org/3/library/asyncio-eventloop.html#asy...

Reply on news.ycombinator.com