On Go, Portability, and System Interfaces (2015) | Hacker News Reader

On Go, Portability, and System Interfaces (2015)

garrett.damore.org

99 pointsAnnatar45 comments

> The problem I believe stems from some misconceptions, and some historical precedents in the Go community.

It's not just "historical precedents in the Go community" it's "ongoing in Go itself": Go's developers will try as hard and as long as they can to perform raw syscalls even on platforms where it's not officially supported, and even on the one platform where it is genuinely actually supported they'll do raw vDSO calls (which are not) which predictably blows up (https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/).

[0] which is pretty much all of them aside from Linux, Golang 1.11 finally stops performing raw syscalls on OSX

It uses ago for system calls on macOS and linux?

I don't know what you're asking.

I presume cgo?

Raw vDSO calls are absolutely supported. The thread you linked is about a Linux kernel that was miscompiled due to a demented Gentoo-hardened toolchain.

Other than buffering, why are libc calls faster than system calls?

libc calls are not faster than system calls.

Explain this comment: https://news.ycombinator.com/item?id=18057649

This comment is saying that the overhead of the libc calls is (or should be) almost nothing compared to the cost of the syscall itself. The libc calls wraps the syscall, it can't be faster (vDSO aside).

Ah! That makes sense now. I thought it would be impossible for LibC to be faster than a syscall short of it cannibalizing the kernel.

> Basically, the Go folks want to minimize external dependencies and the web of failure that can lead to. Fixing that is a goal I heartily agree with. However, we cannot eliminate our dependency on the platform. And using system calls directly is actually worse, because it moves our dependency from something that is stable and defined by standards bodies, to an interface that is undocumented, not portable, and may change at any time.

It's also for performance reasons. Calling a C function from Go is actually quite complicated and can involve copying a lot of data. Using a syscall directly avoids that cost.

That has got to be a total drop in the bucket compared the cost of the syscall itself.

Kernel interrupt handling is incredibly quick.

Tasking switching an the related MMU updates is slower. As you have to perform a kernel interrupt handling action just to start initializing the task switch.

Interrupt handling is quick, paging isn't.

The KPTI Meltdown fixes mean that interrupt handling _is_ paging on a lot of chips out there.

Careful. x86 interrupt handling is slow. System calls use an accelerated mechanism that is much faster.

drops add up

Not when they are 1:1 with what you are talking about, do they?

I mean, I get that across the lifetime of an application, you might see that 1% of your cost is in this somehow, but then the easier response would almost certainly be to do less syscalls, since if this is taking up 1% then the syscalls are taking up 10%+? (And that is assuming this is 10% of the syscall. Ish)

"Using a syscall directly avoids that cost."

Quoting the author of the article,

"That said, this is nothing compared to the cost of a SYSTEM CALL. System calls are measured in usec, usually 10s of them, depending on the complexity of the system call you're calling. (In some cases, such as disk reads, it can even go to milliseconds.) I'd guess on average the cost of the transition is less than 1% of the system call you're making."

The article is not accurate.

Pre-KPTI, the cost of a system call might have been hundreds of cycles total. On a modern chip, that’s ~50ns.

Add kernel code for relevant operation. A read on a pipe might be 500ns total. A read on a cached file would be similar.

Add a few hundred cycles for KPTI.

A typically function call will be tens of cycles, but with lots of copies and stack changes can easily be hundreds.

>How did we wind up in this ugly situation? The problem I believe stems from some misconceptions, and some historical precedents in the Go community. First the Go community has long touted static linking as one of its significant advantages. However, I believe this has been taken too far.

How about people just go out to solve their problems, in their own platforms, and don't care to support all platforms?

It's not like most of those Go projects are big in the first place, and one or a couple of contributors don't have the means to test other platforms much, or the time to write it platform-agnostically.

I think this is likely the reality. I don't think this is necessarily a community problem so much as it is the reality of software development today. Most developers given the time and resources would probably enjoy making their libraries and applications as portable and flexible as possible -- who doesn't love to see their work reused? That being said, on a typical software delivery cycle you optimize for what you think _most_ people are using, and likely what you yourself are using -- standard flavors of Linux operating systems running in one of the big cloud providers. This is not to say Go doesnt have applications outside of this space -- it clearly does.

The only Go apps that I’ve knowingly used are from Hashicorp - Nomad, Consul and Vault and they were all running on Windows.

I found it quite refreshing that the “install” process was copying one relatively small executable.

That being said, Terraform is quite popular and I would think that it’s used by Windows admins/devops locally.

You've also probably used docker, which is written in Go.

Never used Docker. That was the advantage of using Nomad. It can orchestrate anything - Docker containers, executables, Shell scripts, etc.

I mean, afaict nomad uses docker to orchestrate docker containers...

It's not like the author is asking for people to write a bunch of whatever Go's equivalent of IFDEF is. The author's proposed alternative code is just as concise and straightforward, it just has the benefit of using something that's portable across multiple platforms.

However, this created a great ecosystem where everything is usually cross-platform by default. With a lot of projects being developed in Go in the cloud ecosystem, developers can more freely choose the platform they want to develop with. Same story with tools written in rust.

If the goal was "Go is a language for Linux", fine. But that wasn't the goal. From the first release it was targeted to macOS, Windows and other platforms, but implemented in a way that didn't make any sense on those platforms.

Actually Go was implemented as OS X and Linux language initially.

They left up to the community to build up Windows support.

Certain packages are still not available on Windows, like plugins for example.

Easy to find that out searching early gonuts posts.

Ok I stand corrected; they only botched it on 50% of their initial targets :)

Because that's doing a disservice to potential users?

If the Go culture doesn't favor writing portable code, that's absolutely a problem with Go's culture.

Not to mention systems without syscalls [as concept] at all.

Simple example: GNU/Hurd’s glibc works quite differently.

Is there an official documented rationale for why the Go standard library doesn't go use libc, and instead does all this direct syscall stuff?

I've heard various stories about it over the years, so i would appreciate an authoritative source.

My understanding, which is certainly not authoritative, is that Go wants to be 100% statically linked. libc is usually dynamically linked. MUSL is Linux only.

It often does use libc. OP's article mentioned that they're using it more on Darwin now, but they've also been using it on Linux for a long time to do some of their DNS resolution (where implementing a stable parser for /etc/resolv.conf and friends is nobody's idea of a good time).

I don't have an authoritative source, but I think one of the big downsides of CGO for them is that it hurts their cross-compilation story. Cross-compiling a pure Go program is trivial, which is great, but by default it disables CGO.

Exactly, it disables CGO. So if you cross compile Windows -> Linux the pure Go DNS resolution code gets used.

     why the Go standard library doesn't go use libc

They do you use a `libc` they use `musl-libc` [1] as apposed to `gnu-libc` (gblic) [2]. There are some differences [3], the largest (without getting into various API incompatibilities) is licensing. `musl-libc` is BSD licensed, while `gnu-libc` (glibc) is GPL licensed.

The parent post does explain, about as well as any member of the go-team has to date for their usage of `musl-libc` over `gnu-libc` (glibc). Overall `musl-libc` does focus on correctness a lot more then `glibc`, and static linking is a large benefit.

Static linking isn't an issue when you expect applications to be short lived transient items. Where system images only exist for days, or hours before being redeployed by either patches, or CI-Managered triggered updates. I think this is the largest disconnect. Go expects the development pipeline to be quick so nobody expects multi-year lifespan binaries, which is directly contradictory to how many expect executable binaries to behave.

No that I'm authority on this, just stating a common disconnect I've experienced working with ex-Googlers.

[1] https://www.musl-libc.org/

[2] https://www.gnu.org/software/libc/

[3] https://wiki.musl-libc.org/functional-differences-from-glibc...

[4] https://github.com/golang/go/issues/9627

> They do you use a `libc` they use `musl-libc` [1] as apposed to `gnu-libc` (gblic) [2]

No. Historically they don't use a libc at all if you don't use the DNS subsystem. The latest version (1.11) finally started using libSystem on OSX (where performing raw syscalls has never ever been supported, and Go broke several time because they refused to acknowledge that).

My Go v1.11 compiler still emits symbols from `musl-libc` for static builds on linux. I had to dig through on object dump just on friday.

Are you running on a linux distro which uses musl, such as alpine?

Are you running the build inside a docker container which uses musl, such as alpine?

Those are the cases where this might happen.

On a normal linux distro, this will not happen.

This post is just plain wrong. They do not default to musl on any platform.

They only default to the system's libc (which is typically gnu libc, though could be musl) if certain networking constructs are used. They otherwise default to not using any external libc library, but rather implementing one with syscalls internally.

The above is for linux. On other platforms it varies further, but to my knowledge the parent post is not right on any platform.

Performance, most likely. Go wants small stacks, which requires all dependent libraries to be compiled by the Go toolchain, or else suffer performance consequences due to stack switches. libc obviously wasn't compiled with the Go toolchain, so calling it takes the performance hit of stack switching.

I think the key to the rationale is in the name. libc is a library for C. Go doesn't use the same calling conventions as platform C and does a number of things that make it not really play well with C code.

The authors want a world where you can just make a read syscall without using FFI.

It's possible this is a unrealistic expectation. Linux kernel interfaces are generally quite stable. On OSX and windows the kernel interface seems to be an undocumented unsupported API which is sad.

The Windows programming interface comes in the shape of the system DLLs - you load kernel32.dll or whatever, find the symbols you want, and call them. No C necessary, and the interfaces never change. (You might need some stack space to call the functions though.)

Reply on news.ycombinator.com