Hacker News Reader

Comment by useerup | Hacker News Reader

Windows async I/O ("overlapped" in Windows parlance) is backed by a thread pool (completion ports).

Windows does not spin up or allocate a thread just to wait for an IO operation to complete. Rather, when an IO operation completes, a thread is allocated from the thread pool (hence the name completion ports) on which the callback happens.

So there is no excessive allocation of threads. Everything is event-driven and kernel managed until the operation completes.

View on HN

Simultaneously, people overestimate the costs of threads, and underestimate the efficiencies they provide. At extremely high loads, they become an issue, but it's quite rare. And at normal loads, threading is (IME) often simply more efficient than typical async solutions in practice, probably since a thread gets synchronized access to its internal state and consistent ordering of execution (almost) for free, whereas async code needs to (usually) reimplement that in user mode.

People have taken the notion that threads are slow in some circumstances and made the conclusion that threads are slow, period. But that's simply not true.

The rate of computation is obviously very similar for threads and async, but the rate of multiplexing is not. Switching between processes or threads takes more time than switching between fibers/coroutines. So the general observation that for massively multiplexed workloads fibers/coroutines are lower overhead is generally correct, but the more interesting question is of course "what qualifies as a massively multiplexed workload?" ... I'd argue most kinds of web application servers don't.

Yeah, and the devil is really in the details there; not all context switches are created equal. If you're dealing with less than (rough ballpark) 1000 requests per core per second, there's just no way context switching is going to be anywhere near a bottleneck. Depending on all those details (app/OS/processor/lunar cycle), you may be able to deal with 10 to 100 times as many before context switching is an issue.

Those kind of servers are simply quite rare (IME). Nothing wrong with interesting problems in niche spaces, but it's not something you should be worrying about by default.

(No idea what "big" servers provide public stats, but e.g. https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... ) lists a random day for stackoverflow having 209,420,973 http reqests, i.e. a little less than 3000 per second. I doubt context switching is going to matter for them (in the hypothetical world all this was served by one HTTP server, which of course it isn't).

I largely agree with the above and I would say that 1000 reqs/sec is probably a decent threshold for considering when async IO is going to matter for performance. That said, the details of your particular workload may benefit from async IO at significantly lower levels. As an example, one message routing application on which I worked with a typical workload of ~100 reqs/sec increased its performance by about 5x when switched from a blocking thread per request model to async IO. The application typically maintained a larger number (500-1000) of open but usually idle network connections. With that particular workload and on that particular platform, the overhead of thread context switching became a significant factor at much fewer than 1000 reqs/sec. One hint that this was the case was relatively high percentage (30%+) of CPU time spent in kernel mode. Switching to async IO dropped kernel time to about 5% on this particular application.

Yeah, that's a neat case in point (and illustrates nicely that there are cases where async is super-valueable!).

You can do ~ 1 million thread context switches per second, per core. That's a lot of disk operations in flight before the context switch overhead becomes a large component in your performance equation. Multiply that by 10-20 cores for a server CPU, too. If you still haven't pegged your storage hardware IOPS, I'd say you have a "happy problem" :)

the problem with threads is that unless you spawn an ungodly amount of them, you might still not be able to issue enough pending blocking disk IO operations to saturate the disk bandwidth.

I never imagined that the cost of threads was predominantly a performance cost (though obviously that exists too in some parameter regimes).

The real cost of threads is that threaded code is far more difficult to get right than non-threaded code. The closest thing I know of to an excuse for using threads is that we are already using threads, so all the complixity is already there.

After all, once every line of code is already a potential race condtion, it hardly matters if you turn a few more of them into real ones.

Reply on news.ycombinator.com