Hacker News Reader

> You cannot spin up 100K native threads to handle 100K requests that are IO bound (which is very common in a microservice architecture, since requests will very quickly block on requests to other services). Or well, you can, but the native thread context switch overhead is very quickly going to grind the machine to a halt as you grow.

Why? Bigger MySQL servers regularly use 10k native pthreads which spend most of their time waiting on IO.

Context switch overhead isn’t linear with the number of threads! The Linux scheduler runs O(logN). Moving from 10k MySQL threads to 1 thread per core will give you less than 10% extra throughput.

Doing IO means context switching. Goroutines don’t get a free pass here.

There are other costs to regular context switching as opposed to goroutines/greenlets (the green threads that gevent uses). I don't remember the details but specific attention was paid to the point of making context switching and other resource consumption by these green threads cheaper than native threads, so I suggest reading about it in greenlet/Golang docs :) You can also try searching for C10K which was the term people used to discuss how to achieve 10K connections, and is often associated with cooperative threading.

For example, the cost of the context switch itself (storing all registers) is more significant with native threads.

Just try spinning up 100K threads that each print a line and then sleep for 10ms, and see how high your CPU usage gets.

Also, doing IO does not necessarily mean context switching - it means calling into the kernel (system calls). If you use an async IO operation (read/write from a socket) and then continue to the next thread, by the time you're done with all ready threads, you're likely to have some sockets ready to read from again, so you might not context switch at all. Kernel developers are working on even reducing the need for syscalls with io_uring, which is designed to allow you to perform IO without system calls.

Green threads are much cheaper to switch than pthreads, yes. In real applications the difference is far smaller than it was 20 years ago when C10k was challenging. In 2020 you can just open 10k threads and forget about it.

With 100k threads and 100k Goroutines, each doing nothing but waiting on a mutex: pthreads in C takes ~20 microseconds per thread and in Go it’s about ~5 microseconds using Goroutines.

This difference disappears really easily. Parse some JSON and it’ll be gone.

Entering kernel code is the expensive part of context switching so syscalls are very nearly as expensive. Reading from a socket still needs a syscall, even with green threads or asynchronous IO.

The more different bits of IO you do, like in a real web app, the less advantage their is to green threads. This is one reason Rust dropped their M:N threading implementation.

There shouldn't be any 100k hard limit for threads at least in Linux, though you need enough memory for 100k stacks of course. You need to increase some default limits for it though (https://stackoverflow.com/a/26190804)

Assuming a generous(?) 20 kB per thread in stack and other corresponding OS bookkeeping inforation you could have 1k threads in 20 MB, or 1M threads in 20 GB.

Doing 100 Hz timer wakeups and IOs concurrently in 100k threads makes 10 M wakeups/second, that takes a chunk of CPU independent of green / native threads choice. Performance vs kernel threads will depend on the green threads implementation.

Yup. The Linux scheduler wakes threads based on IO events. You don’t end up just cycling through 100k threads all waking and sleeping again.

It's worth noting that the c10k writeup came out 20+ years ago, and those bottlenecks have been addressed both by fixing software bottlenecks and 20 years of semiconductor improvements.

Beam VM (Erlang/Elixir) is another example which copes with the problem very well due to preemption.

To expand on this, by allowing preemption on function calls and simply not providing a loop mechanism, you can guarantee an upper bound on how long a process may need to wait until it can be scheduled.

Guarantee is a strong term. If you have an infinite number of processes to running, there aren't guarantees on how long until it is scheduled.

Similarly, with cooperative multi-tasking as in Gevent, you can manipulate scheduling to try to provide better guarantees about wait times. It's just... you can't ignore the problem.

> You cannot spin up 100K native > threads to handle 100K requests that are IO bound (which is very common in a microservice architecture, since requests will very quickly block on requests to other services)

The Varnish server does that all the time and it is not a problem.

Modern OS handle a large number of threads much better than 10 years ago.

In case of python however, you can not do that due to the GIL and the fact python handle native threading terribly.

From the docs:

> We rarely recommend running with more than 5000 threads. If you seem to need more than 5000 threads, it’s very likely that there is something not quite right about your setup, and you should investigate elsewhere before you increase the maximum value.

Yes, and still 5000 will be more than enough to serve most of production load. Even on high traffic websites.

In NodeJS and python "async/awaits" you don't have to deal with deadlocks and mutexes and other ugly concurrency primitives/problems. The reduction in complexity of the program is huge which is something rachel completely discounts.

I'm not sure discounts. I mostly do firmware in that environment you can use either preemptive or cooperative multitasking. That latter is much less likely to result in corrupted state and results in smaller code size. Which is why I like it. But you pay with latency and task starvation problems as Rachel complains of.

So in another reply some guy pulled off a deadlock in node by creating his own external lock primitive outside of the node stdlib which in itself doesn't contain a single lock primitive. Technically, I'm wrong, it can happen, but only if you go out of your way to do it:

See the conversation here: https://news.ycombinator.com/item?id=22515100

Actually Linux can handle a few million threads just fine - you only need to increase the max pid.

You should just update your reasoning and say you can’t have 100M threads waiting for blocking IO.

Can't a CPU bound task just make itself preemptible in Python by calling some dummy async function, like asyncio.sleep(0)?

Yes, but you shouldn't just do that every time as calling into the event loop has its own processing cost. If you do that on each iteration of the loop, it will probably become much slower.

At a previous job, I wrote a wrapper that would compare the last execution time of the event loop scheduler, and if it was larger than some value, it would yield (by sleeping for 0ms). This was intended for use in loops with varying length, which might be long and could be short depending on some external factors, but yielding on each loop iteration made them slower by an order of magnitude.

IIRC It was sending messages to RabbitMQ, which somewhere down the line is writing to a socket, but not necessarily blocking - if the socket manages to flush its buffer fast enough (faster than the processing code can send messages), writing to it may never block, resulting in a CPU bound loop (since the loop may perform work other than sending).

We didn't want people to have to think too hard about how their loop was going to behave (especially as it might mean reasoning about the internals of a third party library), and so the wrapper was born. If your loop was short enough, all it did was compare ints so the cost was negligible.

Doesn't that mean you have to somehow know when it's been running too long and then yield back control somehow? What if the slow operation is something that's atomic, like multiplying some huge numbers?

Well the example given is decoding JSON. If that is happening in a long loop, you can yield once per loop and be safe. Not all problems are neatly broken apart like that, but in those cases how much of a chance does the server have to not timeout regardless, you know?

Note that once per loop might be too often, but you can just measure how long a loop run typically takes and compare to how soon you want to preempt the task, then yield at the right interval.

Seems like abstractions will bite you there, most will just do cool_library.unmarshall(request) in some variant, those libraries will not have the same method for yielding as you have.

Abstractions are meant to be broken! One could probably work around this problem by adding new functions to cool_library, or modifying existing ones, whose code would be copy-pasted from the library but with some asyncio.sleep(0) calls spliced in strategic places :). For legacy projects, it may make more sense to cheat like this than to rewrite the whole project in a saner tech stack.

Before webworkers this was how you did things in the browser to not get the "page is not responding" popup on computationally expensive operations. Break each big operation into many small operations and step to the next phase using setTimeout(1,..)

Many CPU bound tasks don't have convenient break points that are short enough times apart where doing this would be useful.

Yes, `gevent.sleep()` or `time.sleep()` (if you monkey-patch) will yield control back to the scheduler.

> The problem that's described here - "green" threads being CPU bound for too long and causing other requests to time out is one that is common to anything that uses an event loop and is not unique to gevent. node.js also suffers from this.

Isn't node.js IO native threaded?

I don't know, but I was actually discussing time when the thread is performing pure CPU operations and not doing any IO. In this case the JS code does not yield to the event loop, so any other requests waiting will stall.

This is actually an important property of JavaScript that allows it to work without locks (as long as you do not await, your code runs completely synchronously and will not be interrupted). It's not a flaw, but definitely requires awareness.

Gevent code that doesn't spawn more than one native thread (the one that runs the event loop) also has this property - you don't need locks as long as you do not perform IO. In Python's case it can be more tricky, as you might end up yielding to the event loop by indirectly performing IO when logging or something similar.

In JavaScript, the only case, AFAIK, where this can happen is when you await. Nothing else will cause you to yield to the event loop.

The JS event loop is kind of funny, because tasks have no priority and no preemption and so they're executed first-in first-out. If you want to avoid blocking other code paths for more than a certain length of time you need to change the order you push tasks in[0] and you need to limit execution time in some way, which can get pretty complex to reason about.

[0] by "push tasks" I mean using setTimeout, new Promise(), await, requestAnimationFrame, or similar

Ok, so it is opposite in nodejs? There's a single JS thread but as soon as it performs any async IO ( which is the only IO that one should be performing in node ), that IO creates a thread servicing that IO request. In addition to that, if one is crazy enough to run long computational operations there's setTimeout(), setImmediate() and process.nextTick() and friends.

Though I would say it doing any real compute work inline rather than farming it out to workers is a boneheaded idea regardless of the language used.

AFAIK there is only ever just one thread (that runs user code). setTimeout, etc are all also invoked by the event loop.

I can vaguely recall (but it's 1AM so I didn't verify - take with a grain of salt) that nodejs uses an a thread pool for IO, but I guess that's only for IO for which there is no async API, otherwise that would be wasteful.

I imagine a quick search about the event loop and said thread pool would yield better researched answers :)

Nodejs is single process, single thread and single core. Chrome used to run different tabs as threads of a process but that ended since Spectre and Meltdown. Back to single process since then.

  async IO creates a thread servicing that IO request

Nope. The event is a single FIFO queue implemented with libuv. It differentiates between sync and async function and basically goes around the queue running synchronous functions to completion every time it find one and async ones for a set time before switching to next one.

> Nodejs is single process, single thread and single core.

ps -eLf disagrees:

  $ ps -eLf|grep node
  deploynode     16438 16173 16438  0   11 00:21 pts/6  00:00:01 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16439  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16440  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16441  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16442  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16443  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16444  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16445  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16446  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16447  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16438 16173 16448  0   11 00:21 pts/6    00:00:00 /opt/bin/node node-le-manager-api 3550
  deploynode     16579  8353 16579  0    1 00:27 pts/4    00:00:00 grep node

> Chrome used to run different tabs as threads of a process but that ended since Spectre and Meltdown. Back to single process since then.

Chrome supports several different process models, apparently, but as Spectre/Meltdown (as well as future renderer and UXSS bugs) mitigation it now by default runs a minimum of one process per site/origin (site isolation). See https://www.chromium.org/developers/design-documents/process... and https://www.chromium.org/Home/chromium-security/site-isolati...

There are Linux system calls in wide use that are blocking. Both node.js and Golang use native threads to make the system calls and "park and wait" for the response. The runtimes then take the information from the call and incorporate it back into the user code threads(the one user thread in node.js).

The implementation of libuv under nodejs (and the state of the art python async/await) actually uses a total of three threads. It's a complicated model and it's not even strictly correct to call it an "event loop." I don't completely understand it and very few people do.

From a high level you can think of it as a single thread that can only context switch during an IO call. Otherwise the function has to run and block all the way to termination. <---- This is what rachel is complaining about.

Any good reads on that at hand?

Ok, libuv docs seem just fine. Nevermind :)

As long as my 100k threads are all waiting on io the kernel doesn't need to switch to them.

I get that once you reach full utelization, the best the kernel can do is to make it fair, and that's going to cost. That's not a general statement like "100k threads grinds the machine to a halt".

I have worked on a system that spins up 50k threads on a CPU with 72 threads and it works just fine, it's the google web page fetching system. If that didn't work then Google wouldn't fetch web pages. At all.