Threads, Threads, and More Threads
scorpiosoftware.net
scorpiosoftware.net
Seems like that is an ideal activity for a thread, and in fact why most people use them? Combining all those waits into a single thread/process means also creating a dispatcher which then either spins up a thread to do work (ahem), or blocks all other useful work. If the cost is low (and for thread it usually is), having lots of blocked threads often makes the code easier to write and maintain.
This is where Moore’s Law has brought us: if we want better performance, we need to take advantage of parallelism. Thus more threads.
Threads need to scale with CPU cores, however simply increasing #threads doesn't help. You need to split tasks of your program such that more tasks can be executed in parallel while not adding more overhead than you gain by parallelism.
Then you can distribute those tasks on as many threads as possible (while still being bound by #cores).
It's bound by the slowest shared resource in the system (Amdahl's law). That could be disk I/O, memory bandwidth, or power budget.
Agree, because spawning a new thread and killing a thread carries its own overhead so better to have the thread sleep or be paused by the OS and cpu scheduling.
Its all about getting the balance right, but I have noticed dont net apps run their gui on a single thread instead of a window(s) linked to a thread, like in a multi document interface (MDI) setup.
In an OS like Windows, on "typical" hardware, the amount of memory per thread for stacks and bookkeeping the OS and/or userland allocate is usually negligible. In this context also remember that the reserved stack sizes per thread do not refer to physical memory but virtual memory (or "committed" vs "reserved"). A typical thread usually commits 1 page for the stack and the equivalent of up to 2 pages worth for any bookkeeping[0], etc, which would put the memory committed for the 7412 threads the author mentions in the ballpark of maybe 30-100 MiB of memory actually committed.
The OS scheduler should be good enough to distinguish between runnable/waiting (sleeping) threads, eliminating the need to even look at most threads (which will be in waiting state) until some signal arrives putting them back into a runnable state.
You only really get a problem when you have too many runnable threads (fighting over the available resources) or when too many threads keep rapidly toggling between runnable and waitable, as either case would potentially lead to a lot of expensive context switches (and the former case is an indication you're trying to do too much with the physical resources available).
[0] Tho some things may commit considerably larger amounts of stack space and/or bookkeeping space. E.g. if I remember correctly java8 commits 1MiB of stack space per thread by default, while e.g. java11 merely reserves that 1MiB.
The number of threads should only become a bottleneck if you have less than 0.01% of them idle.
A single thread can then wait for events that would require some processing (IO or timers being the most common) and then it passes that unit of work to the thread pool to be processed. Windows even takes pains to ensure that the last thread to process a unit of work, might then also be used to process the next item in the queue of work to eliminate costly context switching, and invalidation of data/instruction cache.
Thread pools allow the best use of threading by: - limiting the number of threads a process creates/maintains. The thread pool itself can grow or shrink as demand changes. - limiting the number of threads that are in a runnable state by queueing work to the pool. The pool itself will be sized according to number of CPU cores, e.g. 2 * CPU cores.
The main idea behind tread pools is that any thread that's not doing useful work is a waste of OS resources, so they make sure that they are either always fed with work, or get destroyed when idle for too long. From an overall machine efficiency perspective this makes total sense.
Having hundred or even thousands of threads created and idle is massively wasteful and bad design, the OS should not be burdened with such housekeeping.
Note that Apple's GCD also works on the same principle, as does libev/libuv.
This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance, especially on systems with lots of cores. Similarly, there is no requirement to spawn separate threads for task concurrency. The non-blocking coroutine-style concurrency common in high-performance server software supports much higher concurrency and with context switching costs that are much lower.
The thread-per-core coroutine-style is easier to write and maintain too -- the logic is almost entirely single-threaded. No deadlocks, races, etc to worry about because lock graph concurrency is complicated.
It wouldn't. There will only be a context switch if the thread is woken up for some reason (eg. the operation it's waiting on compeltes). If a thread is spending a majority of its time waiting, then it's not going to be doing a lot of context switches.
> This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance
That's a stretch. There are some specific real-time usecases where you want to lock a specific thread to each core, but those are extremely niche and imply total control over what is running on the system.
This architecture has been idiomatic for high-performance server software for a solid decade. It only seems niche if you don't work in an area of software that cares a lot about performance. The performance difference is not small, 10x throughput on the same hardware is not uncommon.
And yes, it implies control of the resources your process nominally owns, though by no means the entire machine necessarily. Again, idiomatic, and software like high-end database engines have been designed this way for a very long time so it isn't a new idea. Designing software that takes full control of its resources, and does its own I/O and process scheduling, isn't magic, just different.
A classic multithreaded software architecture will never be remotely performance competitive with a thread-per-core architecture. It isn't expected in theory, and is never demonstrated in practice. Modern thread-per-core architecture has its origins in supercomputing, specifically to enable better scalability as the number of cores per CPU (and server) became greater than 2.
I’ve spent the last two and a half years mainly programming in single-threaded NodeJS environments with a focus on heavy same-task concurrency, and I’ve still run into deadlocks and races, despite the lack of OS threads involved.
However, your point that context switching is expensive, and is why thread-per-core architectures are so common, still holds.
... no. Not if you rely on blocking system calls. Or wait for anything that isn't the CPU. Or run more than one application per CPU on your system. Or manage concurrent code flows at any level higher than source code.
But now you have more than one application on the machine? Or even, code from multiple vendors which all need to issue I/O concurrently? How do you integrate their work with your custom scheduler?
You need some centralized, system-wide scheduler everyone agrees on.
That's what threads are.
Currently every app that uses tasks has it's own executor with a corresponding thread pool, which may be more efficiently used if it was shared by using a single async executor across all processes.
of course, it requires to agree on one executor which doesn't sound realistic, but would it make sense in general?
But also, this is essentially what threads are... the main advantage of threads being, they coordinate raw assembly code, instead of something interpreted or JITted. OS schedulers are pretty well optimized by this point. The major overheads relate to maintaining security boundaries (e.g. swapping out the TLBs on context switch) and the cost of starting up a thread. Just having threads sitting around doesn't really cost anything.
Erlang gets away with this because of its programming model; it doesn't matter if it's interpreted or JITed or if you managed to compile it ahead of time. The relevant parts of the programming model are that any function call can yield, and because it's a functional language, there's no looping without function calls, so you can't stay busy forever without yielding.
The value of green thread over native threads is the difference in numbers. I haven't tried, but I imagine most OSes would fall over if you did a native thread per connection and you had a million connections; some OSes would probably hit a thread limit before then. Of course, if you've got a million connections, you're probably starting many per second, so startup cost is important (although maybe you could do a pool if it's cheaper to clean up an otherwise dying thread than kill it and start a new one)
Edit: read the article; that was a weird rant; I'm sure there are better things to rant about on an idle windows machine (like how much disk I/O they do... especially on spinning drives with Windows 10)
In Midori all programs were written in a special dialect of C#. Safety would have been ensured by the language, but all programs it actually ran were probably also written by very competent programmers (which doesn't happen in real life ...).
If I understand things correctly, the language would have imposed an early example of "Structured Concurrency" (although that term hadn't been invented yet)
<http://joeduffyblog.com/2015/11/19/asynchronous-everything/>
And it means backtraces in gdb work.
I've worked on several thread-per-core systems, including embedded systems in which each thread owns a core and does not make system calls or context switch. You can do that, and it's ideal performance-wise, but it does not apply to multi-process systems -- how else are they to coordinate workflows, than through kernel-visible concurrency constructs: i.e., threads?
In language implementations with built-in user-mode schedulers like Go and Erlang, you get closer to this ideal for, as they will spin up a thread per core for CPU-intensive workloads. But even there, blocking system calls are still the norm (at least with Linux, which is only very recently moving toward a generic async system call system, with io_uring), requiring many more threads per core to saturate I/O workloads. And that still doesn't address the multi-process coordination problem.
But green threading doesn't address the OP's issue of "too many threads": a process using green threads needs to spawn one thread per CPU if it plans to use the CPU's full power.
On say, a 16-core system like the OP's, you're back to 16 threads per process -- on average, more than the OP was complaining about (15).
7000+ threads does seem like a lot though and agree there is the general lack of caring about resource consumption in all forms, eg Electron apps. At this point having lots of threads seems almost expected by various OSs and something they solve for to reduce the cost of it so the user doesn't have to worry about it.
Isn't this the case for some older cooperative threading OSes? It's possible in theory, anyway.
Of course, there's a lot of tradeoffs between that and just keeping an idle thread around.
Also the author is examining a desktop machine. Of course it will have a bunch of idle processes. In fact I would argue that if the machine was running 100% CPU usage by a few threads all day long the author would be quite upset at the electric bill, fan noise, and the heat generated. Desktop applications are not throughput bound. They are latency bound and having a bunch of threads on standby is normal.
Also, idle threads aren't a problem, even on a system under load. Especially on a desktop.
Also also, there's no global threadpool, which means if you have K programs on N codes and K > N it's a moot point trying to save threads. Any desktop has K > N. So reducing thread usage in all of your processes is unlikely to help.
What does he think is the point of green threads? Fibres?
Why does he think some languages only allow one thread to run at a time but still have threads?
Answer to all these questions is: he’s mistaken about the original purpose of threads.
Seems to not account for the fact that threads will sleep while waiting for I/O.
According to my understanding you don't need additional threads for this.
We've had this type of cpu time sharing far longer than we've had multicore CPUs.
Or perhaps you're using NodeJS-style async callbacks. Guess what happens... Your async read looks like it's just using an event queue, but behind the scenes, NodeJS calls libuv which creates a thread which blocks in read(), and when read() unblocks and returns your callback is added to the event queue.
Perhaps you're using the async/await style with the Promise API for truly async goodness. Guess what happens... Your async call still ends up calling libuv which creates a thread which blocks in read(), and your await waits until libuv's thread read() unblocks and tells the Promise that it's fulfilled.
Maybe you're using shiny io_uring() - finally (it was proposed many times before) a real async system call interface in Linux! Guess what happens... Your lovely async/await calls the pristine goodness of io_uring_enter() which... creates a kernel thread which blocks in read(), and your await waits until the kernel thread's read() unblocks and reports a result.
Even with async/await languages and frameworks, and event-driven frameworks, a blocking thread is used somewhere in the stack to read from a file.
I've measured how many threads to create for optimal throughput of small reads from NVMe storage, and the fastest number of threads varies but it's generally in the tens to low hundreds, and always more than the number of CPU cores, if the data is not already in cache.
This is a complex area to tune adaptively, and Linux itself is unfortunately not ideal for this (it doesn't tell the userspace scheduler about blocked vs preempted I/O threads). Interestingly, reducing the number of CPU cores (using taskset) while keeping the same number of threads sometimes increases I/O throughput.
(Note: The above assumes Linux, not O_DIRECT with libaio, and that data is not in cache when using io_uring).
[My journey was basically being kool-aided on various green thread variants and async and whatnot, only to realize that threads are pretty damn excellent for most use cases, and all alternatives come with subtle drawbacks.]
Do you think that io_uring (or similar) has a ubiquitous future (as opposed to niche – which there is clearly a demand for atm)? Or is combining kernel and user space scheduling is a forever-hard-to-tune problem? Also, do you see any potential improvements to traditional threading/scheduling that could be helpful alleviating some of the resource hogging issues (which are real but over-hyped imo)? Thanks.
Threads as a mechanism for separating out the concerns that can run independent of each other is perfectly fine. While eventloops and thread pools are good, allowing for things like, say, a stats aggregator to not need to share a work queue with an application's work just makes for an easier-to-reason-about system.
I wonder why zero is only "almost" impossible. Don't you always need one thread to be called a process?