Building static binaries with Go on Linux | Hacker News Reader

Building static binaries with Go on Linux

eli.thegreenplace.net

184 pointsingve61 comments

The secret with sqlite is to use "-tags sqlite_omit_load_extension", if you don't use any extension. (which is 99% of the users)

This is explained in https://www.arp242.net/static-go.html

In the specific case of SQLite, you can use it through WASM now [1]. It uses the dependency and Cgo-free Wazero runtime.

Performance so far has been better than the modernc transpile and it’s probably sufficient for a lot of use cases.

[1] https://github.com/ncruces/go-sqlite3

Why not https://pkg.go.dev/modernc.org/sqlite

Performance is likely part of it. The WASM solution looks faster than modernc: https://github.com/cvilsmeier/go-sqlite-bench

Author here.

Faster was a by product. Maintainability was the goal.

API coverage, ergonomics, extensibility all rank higher in my book than performance.

An example I'd like to cite is sqlite-vec. Alex was able to build a Cgo-free version of it, on his own, which works fine with my bindings. This would be much harder to do with modernc.

https://github.com/asg017/sqlite-vec-go-bindings

I'm also adding support for building off the bedrock branch (begin concurrent, wal2). You just build the branch with wasi-sdk, then embed the resulting blob.

> This would be much harder to do with modernc.

https://go.dev/play/p/IztwH9Jl68i

The WASM solution doesn’t rely on a custom libc or transpiler to convert C code to Go. The transpile is an amazing feat of engineering, but it’s hard to debug.

I can wrap my head around the small amount of wrapping the go-sqlite3 WASM library does. If I had to I can maintain that should the maintainer lose interest. I can’t say the same for the modernc transpile. You can also apply the WASM trick to other libraries with much less effort.

And as noted, it seems to be performing better. As the wasm runtime improves it should pull further ahead.

Careful. We (Tailscale) tried to use static Go binaries a year or two ago built with Zig (zig cc) and the SQLite performance was atrocious. It passed all our tests but it didn't survive deploying to prod. It was a very quick (and uneventful) rollback at least.

(Needless to say, we have better load testing tooling now)

I forget the details, but something about the libc allocator used by SQLite-with-Zig-libc being ... not good.

I wonder if it's Zig or musl that's to blame; did you end up statically linking with musl using musl-gcc, or did you forego static linking entirely?

It's well-known that musl is in general much, much slower than glibc. People keep rediscovering that for some reason, likely because they hear of stuff like old echoes of Usenet posts ranting against glob "bloat", not being aware that a lot of what people call bloat is specialization of a lot of performance-sensitive algorithms to leverage SIMD, special-casing, math optimisations, etc. When you check the glibc mailing lists, it's obvious performance is a predominant concern.

I’ve experienced the same performance issues with cgo + a library compiled with zig cc. IIRC it seemed like an issue with the zig tooling not plumbing the optimization flags through the ancient autotools build system for our required dependency. After a while fiddling, we just rolled it back too.

I haven’t tried this in about a year, so maybe the tooling doesn’t have these issues now.

I've heard you can just swap out allocators pretty easily - did something prevent this? Or perhaps its not as straightforward as I've thought...

This sounds like the musl allocator. Using mimalloc or jemalloc would probably fair a lot better.

Is this a problem in distributions that use musl as the system libc (Alpine) ?

Interesting...

I have been building & using a statically compiled tailscale (with CGO) for a while but didn't notice any performance hits. Script: https://github.com/Azathothas/Toolpacks/blob/main/.github/sc...

Producing static PIE binaries is a bigger challenge still.

  $ go build -ldflags '-linkmode external -s -w -extldflags "--static-pie"' -buildmode=pie -tags 'osusergo,netgo,static_build' main.go

For anyone curious to delve into what is this and why: https://www.leviathansecurity.com/blog/aslr-protection-for-s...

Very interesting and well described! If I were to have one nitpick, it would be the use of "Unix" when it's more specific to Linux. Ex. "The libc typically used on Unix systems is glibc" However I'm sure all of the concepts still apply on BSD, Solaris, etc.

Go is also famous for bypassing libc and issuing syscalls directly on quite a few platforms.

I thought I'd read they backed off of that and started using posix as an abstraction layer.

Go used to do raw syscalls on macOS but changed. Same with some BSDs.

Go always did ~libc on Windows (and Solaris) and still does.

Go still does raw syscalls on Linux, as that's a stable ABI.

libc is a UNIX concept, as the OS API surface, as described by POSIX.

On any other no-UNIX derived OS, it is the C compiler standard library, covering only the ISO C specifies.

All the remaining OS services are exposed by other libraries, which in Windows case, the bare minimum is user, kernel and gdi dlls for the Win32 personality.

Systems like IBM i, IBM z, ClearPath MCP, .... also have similar set of libraries, as do non-POSIX RTOS for embedded.

There’s also Filippo Valsorda linking directly to Rust via .a files. https://words.filippo.io/rustgo/

I feel like there's a lot of potential between Go and Cosmopolitan libc. Go itself does not use libc much, as shown in the blog, but some great libraries like SQLite3 need it (unless you use https://pkg.go.dev/modernc.org/sqlite).

The ability to build a single static binary that works on Linux, Mac and Windows using Go would be life changing for the internal tools I develop at work.

> The ability to build a single static binary that works on Linux, Mac and Windows using Go would be life changing for the internal tools I develop at work.

Just curious, life changing in what way? Obviously, 1 is better than 3, but I'm wondering if there is some other interesting reason.

I recall reading about new Macs with ARM chips not supporting static binaries. Is it not true?

That's been Apple's stance on full static linking MacOS in general, many years prior to the move to ARM e.g. https://developer.apple.com/library/archive/qa/qa1118/_index...

You're welcome to ignore it of course, it's just unofficial and a large pain.

>You're welcome to ignore it of course

How do you mean? Like, is it possible to run such binaries on M1? If so I'd really like to know how

You can always disassemble libc and look for the system call numbers used by the syscall assembly instructions. It’s just that these numbers (and associated arguments and return values) are not stable and can and do change upon kernel updates (in which case libc will be updated to keep the libc interface stable). I believe Linux is the only major OS these days to guarantee binary compatibility of the syscall interface.

I know this works on Macs with Intel chips. But the ones with ARM chips just won't execute fully static binaries, and I'm wondering if there's a workaround.

Nope.

Thanks. That sucks

If you can convince Apple to change this code let me know: https://github.com/apple-oss-distributions/xnu/blob/94d3b452...

I’m guessing not. According to man ld on macOS, the -static flag, to produce a fully static executable, is only used to build the kernel. I don’t believe fully-static executables were ever officially supported on macOS, although they would work.

It only works if you don't ever upgrade macOS. Even a patch update sometimes can break it.

For clarity it's not the chip/ARM that causes the limitation, you can recompile the kernel (it's open source) to remove the block and it'll work fine - it's just a ton of work.

Alternatively, Linux :).

All Macs don't support static binaries. That's because the syscall interface on macOS is not stable, only libc is guaranteed to be stable.

The same exact binary working isn't going to happen without runtime performance penalties because the syscall numbers are different on different platforms. Also I believe on windows it's not possible to avoid linking some system libraries to use windows.h stuff, there is no stable ABI.

Linux is the exception among modern OSes to have a stable syscall ABI, everyone else offers only the proper OS API as entry point into OS services.

Once upon at time, static linking was the only thing OSes were capable of, all of them moved away from that, and there is no turning back outside embedded bare metal deployments, just become some people are obsessed with static linking.

Though msvcrt.dll has a stable subset of functions available on all Windows versions.

Hmmm…what about compiling to wasm, and/or then converting the wasm to C?

https://github.com/wasm3/wasm3/releases/tag/v0.4.8

https://github.com/WebAssembly/wabt/tree/main/wasm2c

Has been done and works pretty great: https://github.com/ncruces/go-sqlite3.

Though doesn’t convert the WASM to C, it runs the WASM in Wazero instead.

How does that solve what the person I replied to was asking for?

If I understood you, it doesn't help much, no, but neither does what you suggested.

You're suggesting compiling Go to Wasm (presumably using the wasip1 target?), then converting that to C using wabt, then using Cosmopolitan to create an APE… is that it?

Well, that's not going to work.

First of all, Go's wasip1 target doesn't even support cgo, so if you want SQLite, you're dead right there.

Then, even if you used say TinyGo (which might support cgo, not sure), WASI just isn't a great target to compile SQLite into. WASI is a pretty limited syscall layer. You'd end up with no file locking, no shared memory. Also no threads.

Then, on top of that, you'd layer Cosmopolitan issues. Having written a portable SQLite VFS from scratch, I am not impressed with how they just paper over file locking semantics incompatibilities between OSes, and then confidently ship a forking webserver with SQLite bundled in. It takes a certain temerity, and not running many SQLite torture tests.

Wasm as an intermediate target is great for (single threaded) CPU stuff. WASI is great if you can fit it, but otherwise, it's not, not really.

What's the best way to include the go runtime itself, as in ability to invoke the "go" program from the program itself . I'm not talking about embedding it or downloading it. I want it included within the program.

How are you supposed to include it within the program without somehow "embedding" it? Or am I missing some vital understanding of what "include" vs "embedding" means here?

By embedding it, I mean using the embed feature to pack the golang binary into the program. What I am going after is similar to kubectl and kustomize. The kustomize source code included with kubectl, it's not a binary packed in and extracted

Reply on news.ycombinator.com