Why you can have millions of goroutines but only thousands of Java threads

I would imagine register based VMs have a leg up in this regard?

We think we can achieve the level of performance we want with stack copying.

Since it's Java, I expect it to be slightly faster than Go while using double the amount of memory.

Could you write more about it? Maybe a blog post?

If you write your app with more explicit state rather than implicit state (bound up in the stack), such as by writing a C10K style callback-hell, thread-per-CPU application, you're going to get much much better performance. The reason is that you'll be using less memory per-client, which means fewer cache fills to service any one client, which means less cache pressure, all of which means more performance.

The key point is that thread-per-client applications are just very inefficient in terms of memory use. The reason is that application state gets bound up in large stacks. Whereas if the programmer takes the time to make application state explicit (e.g., as in the context arguments to callbacks, or as in context structures closed over by callback closures) then the program's per-client footprint can shrink greatly.

Writing memory-efficient code is difficult. But it is necessary in order to get decent throughput and scale.

It's true that it's hard for a scheduler to "know" that useful work is going to be done, but that's just a fact of life, just like the CPU doesn't "know" whether or not you're going to use the next page of RAM or go flying off in an effectively random direction.

Imagine that you want to model a quarry. You have a pile of rocks, a rock crusher, a pile of gravel and a whole bunch of trucks. A truck takes a rock from the rock pile. it gives it to the crusher. The crusher works on the rock for a while and outputs some gravel. A truck takes the gravel to the gravel pile. You want to simulate different ways of scheduling the trucks so as to be optimal.

You could write your simulation so that you update every object in your system at every time stamp, but that's really complicated and it also makes it really difficult to collect statistics about processes. Another way is to create a kind of "process" for each action in the system and a "resource" for each thing that could be used in the system. You then write the code for each process as a continuation and you request resources, blocking when they aren't available. Your overall simulation loop is very simple: it runs a single continuation until it requests a resource, and then it blocks. It then runs the next continuation until it requests a resource. Then it blocks. You just keep doing this, unblocking continuations when the resources become available (and all other continuations' simulation clocks are passed its current clock).

The nice thing about this is that you need no concurrency at all -- or you can have a concurrent thread of execution for each continuation. The latter will obviously run faster, but that's not necessarily the point. The point is that the code was dramatically easier to write.

For some kinds of simulation, a million continuations is not unreasonable at all. In fact, for a lot of applications it would be great if you could have a lot more.

Concurrency.

Have you heard of the 10k problem?

Some people think the best way to handle tends of thousands of concurrent clients is to have each run in a lightweight thread (or goroutine, or whatever term for the same basic abstract concept.) This is as opposed to using an event-based system like Node.js does. Maybe they're right or maybe they're wrong, but that's where it's useful.

Lets say an IO call to high latency connection takes 4 seconds. With no concurrency in between that IO call the program just sits and waits for a response. Under concurrency the program can switch to another thread while it's waiting for the IO call to complete.

And people were saying it on platforms other than Linux.

Processes or co-routines are often used more as concurrent objects. That is, they're used to separate data and concerns into separate, isolated memory areas. However, they're also sequential programs in their own right, so the system becomes much easier to reason about. If you have a few OS thread per core, you are responsible for distributing work among those threads, whereas if you have many co-routines per "job" you can let the runtime decide and schedule intelligently.

A concrete example is a web server. Implemented in Erlang or Go you would typically spawn a co-routine for each incoming connection. There you would get isolation and a simple implementation (parsing the request, performing the task, returning response and exiting). A crash or bug handling a specific request would not affect the other clients.

Other great use cases are messaging and communication apps (examples like Rabbit Q or Discord come to mind), concurrent databases, etc.

Sure, it can be handled as a big array where every element is a struct consisting of the ingress and egress socket/fd referencse, read and write buffers, and some saved state label, and then crunching all of it in a big while loop, in C, with epoll. Nginx does this, but that big array can become the bottleneck quickly. (Lock contention, cache line conflicts resulting in poor memory bandwidth utilization, etc.)

But if you have low-overhead async-await, then you don't have to solve the big array problem and you don't have to write that huge state machine thing either, things become easy[-er] to reason about again.

From a virtual memory standpoint, every thread would appear to have a fixed size which can be set when you create it. [0]

From a physical memory standpoint, every thread would appear to have a dynamic size (but bounded by the virtual memory size of course).

[0] http://man7.org/linux/man-pages/man3/pthread_create.3.html

Technically os threads are also simply continuations.

You're basically asking if generalized preemptive multitasking that has to solve every problem equally well is really that much slower than specialized cooperative multitasking that is tailored to specific language?

Of fucking course user level scheduling is going to be superior. Just think about it. Languages like erlang and ponylang dispatch to the scheduler after finishing the execution of every single function. Do you really think that switching to a new thread on every function call is going to be faster than not doing that? Consider that the way you're supposed to do iteration is via recursion which means you will have one function call on every iteration and therefore invoke the scheduler on almost every iteration. The user level scheduler can instantly switch to the next pending actor/goroutine/whatever as if it was just a regular function call. Meanwhile your regular threads will have to switch to the kernel first, now has to load the scheduling tree from main memory because the cache are filled with user level data, then has to make a complex scheduling decision and finally switch back and reload whatever data you just flushed out of the cache.

So yes the article's argument about time wasted on schedule makes a whole lot of sense because not only is the scheduling of preemptive threads itself more expensive, it also incurs extra costs through context switches and emptying the cache.

"The paper contributes Biscuit, a kernel written in Go that implements enough of POSIX (virtual memory, mmap, TCP/IP sockets, a logging file system, poll, etc.) to execute significant applications. Biscuit makes liberal use of Go's HLL features (closures, channels, maps, interfaces, garbage collected heap allocation), which subjectively made programming easier. The most challenging puzzle was handling the possibility of running out of kernel heap memory; Biscuit benefited from the analyzability of Go source to address this challenge."

By my count, Go is a fifth system. (CTSS, Multics, Unix, Plan 9.) It's interesting to observe how the second system effect progresses when the same team has been working on the same problem for 60 years.

    package main
    func main() {
      go println("I ran")
      for {}
    }

    GOMAXPROCS=1 go run main.go

1. Go's scheduler is better at scheduling Go. There are well defined points at which a context switch can occur, and the scheduler is in userland so you never really have expensive context switches.

2. Not sure. I would guess so, if only because it saves a couple context switches.

3. Goroutines are absolutely relevant for both. The stdlib webserver is a high-performance, production-grade implementation and it forks a goroutine for each request, and applications frequently fork more goroutines within a request handler.

4. I assume you're comparing Go channels to pipes? In which case the answer is probably always "faster", but how much faster depends on the size of the thing you're transferring since IPC requires serialization and deserialization while in Go the overhead is just some locking and a pointer copy.

5. Go has dynamically-sized stacks. While you can change stack size on Linux (to use very small stacks and thus spin up more threads), I don't think you can change the size of an individual stack. Plus with Go, you don't have to configure stack size at all; it works out of the box on every platform.

For me, that Go's goroutines are performant is just gravy. I like that they're a platform agnostic concurrency abstraction. I don't have to deal with sync vs async ("What color is your function"; which incidentally cost me the last 1.5 workdays tracking down a performance bug in Python) or posix threads vs Windows threads.

They are definitely not more performant than native threads with a work-stealing approach (producer - consumer pattern) that is tuned to the number of cores available. How could they be more performant? Even the simplest switching adds some cost, from a performance perspective it would never make sense to run thousands of threads on 8 cores, whether they are scheduled by the OS or by the language implementation.

So to answer your 3rd question: short lived I/O tasks.

Are you asking to compare mutexes and channels? The article goes through multiple reasons why the go scheduler has more information about how something should execute than the Linux kernel. Reread and then maybe rephrase?

Now you want to scale, so you need to abandon the thread-per-client, everything-is-synchronous-looking model and start doing async I/O the hard way (perhaps with await, perhaps as callback hell). That's a huge refactor / re-write exercise.

But you'd be surprised how many thread-per-client apps exist. It's so easy to write them...

Threads are quite useful in many situations and have great performance characteristics. It comes down to people knowing the language they are using and using it correctly.