Hacker News Reader

Comment by lxe | Hacker News Reader

Thread leaks, which happen more frequently due to threaded async abstractions, such as the goroutine, are less often discussed than memory or CPU leaks, but are much more dangerous in multi-tenant container environments.

A thread leak can lock up your entire node, including all the control plane processes. A container spec doesn't provide an easy way to control thread/nproc/ulimit limits (you can still do it, but it's not straightforward), which in turns leaves pretty much every k8s deployment misconfigured and vulnerable to thread leaks.

View on HN

Goroutines in a single process map onto a fixed number of threads. Even if you have goroutine leaks, you should not have thread leaks. Your program may deadlock or run out of memory, but it will not take the whole system down (at least, not in this way).

> Goroutines in a single process map onto a fixed number of threads

Not necessarilly true if you're using cgo

I'd love to find a deep dive on how goroutines and cgo actually interact.

As I understand it, a cgo function call yields the caller goroutine and then runs the C code directly on the underlying thread, blocking it from use by goroutines. When the function returns, the thread is freed up, and goroutines including the caller can be scheduled again. I'm not sure if the caller is guaranteed to run next, or if other goroutines can crowd it out. I would imagine it probably does run first, if only to receive the return values from the C function and then release its thread affinity. This whole process is notable for introducing some overhead to cgo calls which can be significant if cgo is used frequently.

While you can create new threads in C and thus create thread leaks that way, I don't think any of those threads will be used by the goroutine scheduler, which sticks with the pool of threads it manages.

EDIT: reading the runtime docs, it seems that GOMAXPROCS is not as hard of a limit as I thought:

> The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit. This package's GOMAXPROCS function queries and changes the limit.

I think cgo calls count as "system calls on behalf of Go code" for this purpose. Thus if you have GOMAXPROCS=1 and more than one goroutine and you make a cgo call from one of them, the scheduler may create a new thread so the other goroutines can still run. You don't need cgo to do this though, syscalls (explicitly or through Go's stdlib) can exhibit the same behavior.

So I think it is possible to leak threads this way, but to do so you would need to spawn goroutines calling cgo faster than the C code can return.

My #1 complaint about Rust is that leaking a future is safe. It means the compiler can’t check for async coroutine leaks, and it breaks the borrow checker’s ability to say “nothing else has a reference to this any more”.

Anyway, we’re using golang for some stuff at work, and holy crap, I forgot how terrible it was to work in high level languages that don’t statically check for correct synchronization.

If C++-style concurrency is like a chainsaw, then golang concurrency is like a chainsaw in a bouncy castle.

> My #1 complaint about Rust is that leaking a future is safe.

There’s no other way given the leakpocalypse decision. You’d need an entirely new leak-proof language to fix that, and that means you need alternatives for Rc and Arc (or a way to prevent them creating a cycle).

I'd gladly give up Rc and Arc if it meant the borrow checker treated spawn() like a normal function call.

All the code I write is async, so the borrow checker is effectively broken for me. (Wrapping everything in Arc creates weird false sharing at runtime, and I don't want to spend time debugging that class of performance nonsense.)

For what it is worth, I totally agree. I programmed a fairly large project only to find that goroutines were mysteriously blocking in extremely hard-to-debug subtle ways. So, I had to rewrite a huge chunk. I think goroutines are like Ackbar "It's a trap" trap.

I was liberal in usage with goroutines and channels. But after that experience, I decided to religiously track each goroutine in my head. If I reached the point of not being able to mentally map all goroutines running, I would cut out go. I started using callbacks more too to avoid the pernicious blocking I have experienced.

> My #1 complaint about Rust is that leaking a future is safe.

Do you mean futures that aren't polled to completion, tasks that aren't joined, or literal memory leaks that happen to own futures?

The first and third thing. (They're basically equivalent.)

You can start polling then do std::mem::forget on the future. At that point, the borrow checker thinks the future no longer exists. So, it is unsound to pass a reference with a bounded lifetime into a future (which is why you need all the references to be 'static if you pass something into a spawn, or you need to spray Arc everywhere).

They're very different, because a future may be dropped before it is polled to completion (aka, cancellation). If you leak it, then the drop handlers are not called, and this can affect program correctness.

It is not unsound for a future to own a reference (in fact it's super common - how else would asnc methods work?). If you leak the future, it can't be polled and it can't be dropped, so any references will never be dereferenced. But also that's a pretty contrived example.

Like you could call std::mem::forget on a future that owns a tokio::sync::MutexGuard and then you'd have some problems with deadlock... but that's not an async issue, it's always incorrect to leak an RAII guard (same as if you leaked std::sync::MutexGuard)

tokio::spawn has a 'static bound because it simplifies things, not because there's some fundamental limitation of futures owning references.

What is a "correct synchronization"?

Synchronization such as the whole program is well-defined according to some memory model.

Which is the case for Go:

https://go.dev/ref/mem

A quote from your link:

> programmers are strongly encouraged to use appropriate synchronization to avoid data races

Any time you need to "encourage" programmers to do the right thing, you have already failed in your language design.

And I think OP agrees with me here. OP says "static checking of correct synchronization" which is irresponsibly absent from Go.

So it's absent from every language but Rust?

Javascript is even better, by not having multithreading at all. I am not joking, Go is much worse than Java and C#, but Javascript and Rust are the only mainstream languages where I've seen non-experts reliably write correct concurrent code. Maybe it's true for langauges like Elixir as well, but I haven't tried.

My experience of JavaScript is nowhere near that good.

While it does not have parallelism, the concurrency is as unrestricted as it is in go, forgetting awaits is a very common issue leading to wild tasks running unbounded and unchecked, and when you need any other synchronisation mechanism you have to write them yourself and they’re tricky indeed.

$dayjob’s JS team keeps chasing concurrency issues, and breaking builds because of them.

It's 95% true for Elixir as well; it's very difficult to mess up parallelization or just leave a system in an incoherent state with it. Immutability and message passing help a lot.

I don't know of any other multithreaded systems programming languages that check for data race freedom at compile time.

Single threaded languages are usually data race free. I imagine some multithreaded, purely functional languages are too (everything is immutable, and therefore cannot be modified in race with reads). Of course, SQL running in strictly serializable mode is too.

Of those, the only one of those that's an appropriate choice for systems software development is Rust. The Core C++ Guidelines are a runner up in my opinion: They dictate a subset of C++ that is safer, with the goal of backporting the Rust memory safety properties to C++. Swift has also done a lot in this space.

Alternatively a language can choose not to expose anything low level enough to cause synchronization issues and only make available high level APIs that are correct by construction.

Go does neither. That's why there's this thread on HN that I bookmarked: https://news.ycombinator.com/item?id=31698503

Isn't this addressed by CPU requests/limits + pid limiting?

https://kubernetes.io/docs/concepts/policy/pid-limiting/

It's just more annoying to set up and isn't as widely known, and doesn't work on a per-pod basis.

> PID limiting is a an important sibling to compute resource requests and limits. However, you specify it in a different way: rather than defining a Pod's resource limit in the .spec for a Pod, you configure the limit as a setting on the kubelet. Pod-defined PID limits are not currently supported.

Reply on news.ycombinator.com