Hacker News Reader

Comment by simias | Hacker News Reader

Whenever the topic of Go bypassing the libc was brought up before the concerns were usually dismissed by the Go devs, saying that it was technically impractical due to some of Go's requirements not matching the libc's semantics.

Does this mean that the Go developers managed to work around this problem or that it was just a flimsy post-hoc justification for chronic NIH syndrome?

TFA itself links to another blog post discussing "Some reasons for Go to not make system calls through the standard C library"[1] but as far as I can tell it doesn't explain why it suddenly stopped being a problem on OpenBSD.

[1] https://utcc.utoronto.ca/~cks/space/blog/programming/GoCLibr...

View on HN

> Does this mean that the Go developers managed to work around this problem or that it was just a flimsy post-hoc justification for chronic NIH syndrome?

No, there is almost certainly a bunch of headache having to do system calls through libc for Go. It didn't stop being a problem, there just isn't another option in OpenBSD's case. UNIXs are tricky for anyone who doesn't want to use libc since they typically define that as the official interface to the system. Linux is the outlier here by keeping its syscall ABI very stable.

I understand that but if they're going to go down that route for OpenBSD why not bring it to all platforms? While I can believe that going the raw syscall route can be easier than dealing with libc idiosyncrasies, it seems to me that maintaining both interfaces depending on the target OS would be trickier than just cutting your losses and going the libc route everywhere.

After all as far I can tell it's not just an OpenBSD problem since famously they got breakage in MacOS as well.

I must admit that I haven't taken the time to analyze in depth the pros and cons here, but Go's history of NIH coupled with the fact that basically every other mainstream language manages to work by binding the libc leaves me very perplex.

In particular some of the points raised in the blogpost I linked seem fishy to me. For instance errno being a global: this is in no way a Go-specific problem (multithreaded C couldn't run concurrent syscalls if it was true). In practice errno is thread-local instead of being a true global. It's explicitly documented in the man page:

> errno is defined by the ISO C standard to be a modifiable lvalue of type int, and must not be explicitly declared; errno may be a macro. errno is thread-local; setting it in one thread does not affect its value in any other thread.

I can believe that there are other issues I fail to consider, but again it works for everybody else, what makes Go so special here?

> I understand that but if they're going to go down that route for OpenBSD why not bring it to all platforms?

Because they really don’t want to, so they will avoid doing it until forced to, as previously happened with e.g. macOS.

> I must admit that I haven't taken the time to analyze in depth the pros and cons here, but Go's history of NIH coupled with the fact that basically every other mainstream language manages to work by binding the libc leaves me very perplex.

One of the issues Go has is it uses its own non-C stack. Libcs generally assume a C stack and don’t understand growable movable stacks so they can get very cross when called with an unexpectedly small stack (iirc Go defaults to 2k while the smallest C stack I know of is 64k on old OpenBSD, then macOS’s 512k for the non-main threads).

Obviously I'm missing something here, because I was under the impression that, regardless of how big the stack is, the pages are going to be mapped (in the data structure pointed to by the CR3 register) when accessed and no earlier. Unless the allocation of the stack makes sure that it is mapped in its entirety upfront and stack access cannot incur minor page faults, which may not be so outlandish come to think of it. Can you please provide some clarification here? Thanks!

I'm not sure what clarification I could provide given I don't know what you're missing.

You're correct (AFAIK anyway) that stack pages are mapped lazily. That doesn't change the fact that C stacks are large allocations, while Go will only allocate a very small stack (2k last I checked) per goroutine.

Goroutines don't map directly to OS threads though, do they? So in practice it only matters for "hard" threads. I have no idea about what the Go scheduler looks like though, so I can't say if there's a direct relation between the two (coroutine stack vs. thread stack).

Also the default pthread stack size is (usually) 2MB which is indeed non negligible if you have a ton of threads, but with pthread_attr_setstacksize you can lower it. I can't seem to find the actual minimal size at the moment but I have a vague memory that you can reduce it to 16kB portably.

I'm currently working on a multithreaded Rust program with a bunch of threads and strong memory constraints and I use the runtime to reduce the stack to 64kB, it seems to work just fine.

And again, given that the language is garbage collected I can't help but find it a bit amusing that they're being stingy with a few MB of VMEM per thread. I guess that frees virtual memory for use in the balasts!

> Goroutines don't map directly to OS threads though, do they? So in practice it only matters for "hard" threads.

It matters for goroutines because that means you can't straight call into foreign code from a goroutine's stack, which is why cgo and friends are so problematic.

> Also the default pthread stack size is (usually) 2MB which is indeed non negligible if you have a ton of threads

That's not relevant because it's not vmem is the point. The actual resident size of a 2MB stack is 4k unless you start dirtying more pages. Unless you're running on 32b, VMEM doesn't really matter.

Yes, sorry, let me be more clear, I don't understand why Go should allocate a small 2k stack (you are correct, the Go stack size has fluctuated through the years, nowadays it's 2k) and if it grows go to the trouble of copying it to a larger structure? Instead of just allocating a big stack that will grow lazily anyway?

Obviously the folks that created Go aren't stupid, far from it, so there must be a real and valid reason. But can't easily imagine what it is.

Go is very, very profligate with threads.

When you have 100,000 threads, big stacks add up.

It's only vmem though is the point. It doesn't really matter that you have 100000 stacks of 8MB when all of that is vmem and you've got a page committed on each.

Each allocation also needs an entry in the page table (actually a virtual memory tree). On x86/amd64 the fanout is about 1:512.

If you have 8MiB stacks, then a minimally allocated stack uses a 4KiB data page, but also 2MiB of address range uses up a full 4KiB bottom level page table, and 8MiB range takes up 4/512 x 4KB of 2nd level page table and so on. So you use about 8.03 KiB RAM if you never touch more than 4KB and your 8MB reservations are mostly grouped. Some architectures have bigger pages, increasing fanout but also the minimum allocation.

Contrast to 2KiB stacks without reservations/overcommit - you use about 2KiB of usable RAM + (1/2) x (1/512) of 4KiB 1st level page table + .. , assuming allocation are again mostly grouped. Hence, for up to 2KiB of stack memory you need about¹ 2.005 KiB of RAM. Works the same for 16 KiB and even 64 KiB page sizes.

100000 * (8MiB reserved, 1..4KiB used stack) needs ~784MiB RAM.

100000 * (2KiB reserved, 1..2KiB used stack) needs ~200MiB RAM.

Note that, if you actually touch your reserved stack, even once, your allocation can balloon to possibly tens or hundreds of GiB (100k * 8MiB = ~800GiB), unless you do complicated cleanup, while a segmented stack can keep the allocations within reasonable efficiency, freeing any excessive stack allocations in userspace.

¹ ignoring bookkeeping overhead in both cases, to keep the calculation clear. Hopefully it isn't more than a dozen bytes or so.

> 100000 * (8MiB reserved, 1..4KiB used stack) needs ~784MiB RAM.

Yeah so nothing really relevant, that's 1/20th of a relatively basic dev laptop's memory for 100k threads.

And of course that's an insane worst-case scenario of 8MB stacks, which the linux devs picked because they wanted to add a limit to the stack but didn't really care for having one, Windows uses 1MB stacks and macOS uses 512k off-main stacks, so you don't need anywhere near 8MB to get C-compatible stack sizes.

> It's only vmem though is the point

With 8M/4M/2M/1M/512k stack size, 4K page size, you get about 800M/800M/800M/600M/500M RAM usage, of which only 400M is usable, rest is overhead. At 16K page size, it's ~1600M. Compared to 200M if using 2K side-by-side stacks, in any configuration.

Yes, probably not too excessive, even though it is noticeable when you spawn theads for anything and everything, and run more than a single application on a non-SV-developer PC.

I think main problems start when you actually touch more than the base allocation of the stack (or just use 16K pages). Maybe segmented stacks grow from 200M to 300M, 500M or whatever you actually use at a given time (with say 20-60% efficiency), but your C-compatible stacks might go from 500M to 3G¹ if you on average touch just 32K of stack per thread with some unlucky function, even though at any given time only the same ~100M of memory actually stores useful information.

¹ or more, no idea how high it typically goes, but that in itself is a nasty gotcha, and a likely reason you won't find "lightweight" threading in combination with native per-thread stacks

> With 8M/4M/2M/1M/512k stack size, 4K page size, you get about 800M/800M/800M/600M/500M RAM usage

Simulating this (by creating 100k maps of the relevant sizes) there is no difference in RES between 8M pages and 512K pages: it was ~495M for both, only the VMEM varied (respectively ~780G and ~50G). Touching the second page increased the RES of both to ~816M, which is about what you'd expect.

This is on a more or less stock x64 Mint.

> I think main problems start when you actually touch more than the base allocation of the stack

The thing is you're unlikely to do that in all of your 100k routines, most of them will not grow beyond their first page, and maybe their second… at which point the routine's stack would have grown to 8k anyway.

> maybe segmented stacks grow from 200M to 300M, 500M or whatever you actually use at a given time (with say 20-60% efficiency)

Go has not used segmented stacks since 1.3.

> your C-compatible stacks might go from 500M to 3G¹ if you on average touch just 32K of stack per thread with some unlucky function

Your not-C-compatible stacks will do the exact same thing. Since 1.3, stacks are realloc'd and double in size on every overflow.

The Go runtime will also shrink stacks if able (halving them) during a GC run, but you can do essentially the same thing on your C stack using madvise(2), and without the need to copy stack data around.

So what gain there is, is really only for goroutines which never grow beyond their initial size (and only since the default stack was decreased from 8k to 2k in 1.4), at the cost of all the C incompatiblity mess. And it assumes these 2k stacks are allocated from a reusable pool (which is probably a fair assumption though I certainly did not check it) otherwise they'd be on different pages anyway.

> Simulating this .. there is no difference in RES between 8M pages and 512K pages

Linux memory management has notoriously complicated reporting. Your program only has 495M of usable memory mapped (RSS; not sure where the extra 100M is coming from), but RSS does not count page tables.

You cannot actually use the 400M of sparsely allocated memory (4K at 2+MB intervals) without another 400M of page tables. I'd suggest you try allocating and using >50% of RAM, or just compare how much can you use sparsely vs. dense before your program hits OOM. Note, you may need to enable overcommit, increase maximum map count, if you are mapping region seperately and preferably disable swap to avoid thrashing.

  sysctl vm.overcommit_memory=1
  sysctl vm.max_map_count=10000000
  swapoff -a

You can also watch /proc/memory PageTables total, which should show the difference.

That being said, I don't use go, I was merely pointing out that virtual memory management is not magical, and it has real memory costs when doing sparse allocations (also has quite significant other costs).

> The Go runtime will also shrink stacks if able (halving them) during a GC run, but you can do essentially the same thing on your C stack using madvise(2), and without the need to copy stack data around.

While you could do that, I believe it requires a syscall per-stack, which might be more expensive than a bit of copying, and I'm not completely sure how easy it would be to determine whether the memory has in fact been allocated and needs freeing.

This will mean a virtual memory allocation of 800 GB. Such large allocations can fail, even if virtual, depending on over commit settings.

> Such large allocations can fail, even if virtual, depending on over commit settings.

Given how Go has no issue being prescriptive as hell on other things, I don't see why they couldn't just go "set vm. overcommit_memory to 1 and fuck off", that's exactly what e.g. Redis tells you.

Because that is very OS specific.

AFAIK BSDs don't have a concept of overcommit because you can't even turn it off.

And again it would hardly be the first time Go makes OS-specific decisions.

You aren't thinking with big numbers. Small amounts of admin work add up. Virtual pages aren't free. Go has had to think very hard about what to pare down to avoid dragging to a halt with orders of magnitude fewer threads.

My point exactly, thanks.

Go wants low overhead multithreading and static linkage. Glibc is inherently dynamically linked, and sets up threads in ways Go doesn't want.

Error doesn't work for everyone else. It's a hack, you wouldn't do it if it wasn't for backwards compatability.

But why does it want to not dynamically link against a platform's libc?

What is the advantage in that? including that into a binary seems about as senseless to me as including the entire kernel in it.

Not all platforms have a "libc". Linux is an example: there is no standard c library and while glibc is very common, there are several distributions that use other C libraries like uClibc and musl. For example a gaming handheld i have which is running Linux uses uClibc

Another example is Windows, the platform API does not provide a C library (even MSVC has its own). While there is an MSVCRT.DLL it is not recommended to link against it as it is there only because some other software relies on it and its semantics are for around Visual C++ 4 (IIRC).

Windows used to be an example like any other non-POSIX OS, however as of Windows 10, the C standard library is part of the OS.

https://devblogs.microsoft.com/cppblog/introducing-the-unive...

Ah yes, i've forgotten about it. I wonder how reliable that is as i noticed a bunch of applications i have ship their own copy of ucrtbase.dll.

But Linux is not a platform in general.

It is already next to impossible to write software that requires “Linux” and nothing more with all the kernel functionality that can be enabled or disabled.

Linux is a component of many different platforms, which indeed do provide different libcs, but also different t.l.s. libraries, different c.p.u. architectures, different Linux configurations and whatever else.

As far as I know with respect to Windows, it only provides stable interfaces viā C libraries, and does not have a stable binary interface to the kernel directly.

Right, Windows has a stable C API, but the comment i replied to was about a platform libc.

For containers Go static binaries are great. Used to build Docker containers for Kubernetes using a 8 MB Go binary. That was all you needed. No libraries. No "minimum OS' Alpine or Ubuntu image.

That seems to be a sensible use yes, but those aren't really any targeted and supported platform.

Not every libc is a glibc.

True but errorno is still a hack.

> > [errno]

> I can believe that there are other issues I fail to consider, but again it works for everybody else, what makes Go so special here?

errno being thread-local doesn't really help all that much with a M:N threading model -- the runtime is going to have to be extremely careful to not stomp all over it. (Whenever calling into anything which uses errno. Of course system calls do that as well, but it's a much smaller surface area than "most of libc".)

It's OpenBSD's plan to eventually shut down any other option via system call origin verification. So golang has to bite the bullet at some point.

"The eventual goal would be to disallow system calls from anywhere but the region mapped for libc"

https://lwn.net/Articles/806776/

One reason not to go through libc is the possibility to create a pure Go userland like it’s done with the u-root project that is apparently in use at Google and resembles Busybox based systems with a memory safe language.

Unless you plan to never run anything but Go on your system you're going to need a libc at one point or an other. If anything this decision reinforces the NIH hypothesis for me. Almost like a prideful "NO C ALLOWED" stance which is not really pragmatic IMO.

The only counterargument I can think off would be to remove the libc footprint for ultra-small, go-only distributions but I highly doubt that it's significant on any modern system (even an embedded one) and if you care so much about reducing the runtime footprint why would you go with a GC language in the first place?

>> Unless you plan to never run anything but Go on your system

That's actually exactly the use case for a lot of containerized applications written in Go.

edit: I've worked on multiple applications like this and there's the odd service where you need some other dependency, and then that service has to start running on Alpine Linux or something. But if the majority of your many "microservices" are pulling in bits of userland from a distro, they stop being "micro" very quickly which is a big deal when there's a lot of them. It's far preferable in that architecture if the entirety of your userland is your own binary.

>Alpine Linux

From their website:[0] >"Alpine Linux is built around musl libc and busybox"

I could be off-base here, but it seems to me the better analogy would be some sort of "Go-Linux" that's like a Docker linux OS written entirely in Go.

[0]: https://alpinelinux.org/about/

I wasn't making an analogy - I literally use Alpine Linux when I need some common *nix tools but I want something lighter-weight than the Docker images from more mainstream distros. musl libc and glibc aren't 2 fundamentally different things - the former is just a more permissive license and a lighterweight implementation.

In containers it’s entirely reasonable and desirable that you only ship your app with minimal dependencies. I suspect that the people who are grumpy about upstarts doing away with libc are also inclined to argue that containers are the work of the devil or some such.

>> containers are the work of the devil or some such

And ironically it was the BSDs and Solarises of the world that had more fully developed similar concepts long before they were mainstream on Linux. I'm not terribly surprised by OpenBSD's stance here, though, and even though I'm more in the Go / Linux ecosystem I can't really argue - it's an unfortunate collision of very different philosophies.

I've actually wondered if there would be demand for a Busbybox / Toybox clone built in Go for this reason. I have zero need for it personally, but I think it would be fun to make.

> as far as I can tell it doesn't explain why it suddenly stopped being a problem on OpenBSD.

It doesn't. It's just that as previously with macOS, Illumos, or Windows, the Go project ended up with its back against the wall: in this case, ultimately only the OpenBSD libc will be allowed to make syscalls so their choices are "use libc" and "no Go on OpenBSD".

And in all honesty the followup https://utcc.utoronto.ca/~cks/space/blog/unix/CLibraryAPIReq... is much more convincing as to why it's a hassle to go through libc.

Normally I agree with anit-NIH sentiments, but why are we all supposed to just have to be happy with a language ABI that's been around since the 1970s?

You would probably find yourself asking similar questions a lot in OpenBSD land - they're very committed to stability and security even if it means forfeiting many modern features. As I say in another comment, wanting to run Go on OpenBSD the way it does on Linux is a collision of very different philosophies. Hard to say one is wrong, but if they want to coexist one of them would eventually have to compromise, and it was never going to be OpenBSD in the end.

If it ain't broke...

Having a common entrypoint into the kernel we can hook into is fairly valuable IMO, especially when said interface is for the most part stable between all unx (and to a somewhat lesser extent even on Windows).

Having this lowest common denominator ABI makes interop relatively* painless. But clearly Go wants to eat the world and doesn't seem to care very deeply about this. I suppose it's a valid approach, even if I don't necessarily agree with it.

  If it ain't broke...

user supplied read addresses

getpwent() - actually really the whole notion of users

file permissions

aynch storage event completion (all asynchrony is pretty broken)

system metadata control (sysctl, /proc, interface socket messages...)

signals

ttys

errno

affinity interfaces ...

Could you give any pointers to what is broken about these, and how a better way to do it would be? (I'm thinking about implementing a small system call library interface and I'd like to not repeat the mistakes of the past)

user supplied read addresses - when a process does a write() it makes sense for it to supply the data as it filled it in. but it almost never makes the same sense for a read(), and using the user address requires a copyout. i'm convinced that having the read() results show up in a kernel-allocated buffer in userspace is a better idea, but this is somewhat subjective

getpwent(), the whole notion of users - we dont use computers the same way we used to in the 70s. talk, write, finger, wall - they aren't very fun anymore since its either just me on my laptop, or one of the 100s of virtual machines floating around. more importantly, the attempts to glue unix system user identity to distributed identities (PAM) have really turned out to be a mess

filesystem permissions - these are clearly insufficient given the number of system-specific addons here.

signals are so riddled with constraints and incompatibilities that they are basically useless - except you have to fiddle with them for things like SIGPIPE

ttys were already kind of broken when they were relevant,

errno is actually a property of the libc, but the status interface is pretty broken - have you even grepped the kernel code to find out what might be issuing an EINVAL?

why dont you mail me at yuri tenuki org? i love these kinds of chats

The NIH one, as always with Go

It's a clash of two NIH worlds: Go and OpenBSD. :-)

I mean OpenBSD is hardly the first let alone only system which mandates going through libc (or equivalent) to interact with the OS.

Don't get me wrong, it makes perfect sense. And Go seems to aim more towards pragmaticism than OpenBSD's elegance and correctness.

OpenBSD is deciding to invent their own paintbrush without even looking at what other people are painting with, while Go took the closest bucket of paint and threw it on the floor, not realizing they were standing in a corner while doing so.

And I say that as a happy user of both.

> Go devs, saying that it was technically impractical due to some of Go's requirements not matching the libc's semantics.

I've always considered claims like this to be bullshit. I've worked a lot with both libc level system calls and (for crash report generation) direct system calls bypassing libc. They're the same interface. There's nothing you can do with a raw system call interface that you can't do through raw libc --- with rare exceptions for things like libc not having system call wrappers for some new system call or legacy ill-advised emulation like in fallocate. None of these exceptions is a justification for bypassing libc for calling open(2) or write(2).

As far as I can tell, the real reason Go eschewed libc system call interfaces is that the Go developers want to think of themselves as "not C".

Go has a different calling convention and stack layout from C, and wanted completely static binaries which you cannot always get when linking to the standard libc implementations. What is "bullshit" about these compatibility concerns? It's not like _Ken Thompson_ was unaware of how to use libc.

There are many languages and runtimes that use their own calling conventions - not the least of which is C++ - and even stack layouts. The problem with Go is specifically the spaghetti stack that they need for coroutines. But that is an inherent design issue - a systems programming language is supposed to play well with the system, and that means being able to use the standard OS APIs properly. Which on basically all platforms other than Linux means going via libc or equivalent (e.g. Win32). Go designers just unilaterally decided that the rules don't apply to them, because reasons. And the end result was stuff like this:

https://github.com/golang/go/issues/16606

So now they're gradually backtracking on all platforms - macOS switched to libc a while ago.

Yet Go can call libc functions just fine on many OSes. Why can't it on Linux? Every single other managed runtime --- Mono, Java, Python --- can call through libc just fine. There is zero technical case for Go not doing the same thing.

The bullshit lies in these FUDlike insinuations that using libc would limit Go in some way. These insinuations are never backed up with technical specifics. I don't care that famous names are involved with Go: the presence of these people doesn't make Go's behavior correct or necessary.

There is zero technical case for Go doing what it does on Linux. You can make a "fully static binary" (which is a terrible idea anyway) with libc. Nobody should be making static binaries.

Those sound like assumptions. Can you back it up with examples of what cases Go developers saw as problematic and point out what they should have done instead (and that this indeed does not represent a problem)?

Reply on news.ycombinator.com