Hacker News Reader

Comment by acqq | Hacker News Reader

I'd like to read what the actual benefits of not linking to C sqlite implementation are. I can imagine using wazero.io for libraries, but I fail to understand _why_ would anybody prefer to run sqlite that way from their own go program, except if it's in cases where execution times are clearly not important (which explains to me the background story).

View on HN

The major one is (cross) compilation and portability.

You can (e.g.) easily whip up a small CLI that embeds an SQLite database in the binary [1], cross compile this to all the platforms Go supports [2], copy one file, and be reasonably sure it'll run in any of them.

This is an example of embedding a database in a binary (with go:embed), and opening it for reading/writing in memory. You can then save the modified copy to disk using the backup API:

[1]: https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/memdb#e...

[2]: https://gist.github.com/asukakenji/f15ba7e588ac42795f421b48b...

I'm still missing some details to understand that story, as I don't believe that there is any platform that Go supports on which C sqlite would not run, more the opposite, the memory requirements of wazero.io way must be clearly higher than of the C sqlite library?

It seems to me it's not about would C sqlite run or not on that platform but that there is still some other convenience, and that's what I'd still like to learn, as I'm sure ncruces does solve some specific problem.

To compile C for a given platform, you need a C compiler for that platform. Which definitely exists but you may not have it at hand.

To cross compile from your platform A to that platform B, you need a cross compiler, which may be even harder to come by.

If you're working on a Go project, you already have a Go compiler that can do this (cross compilation) very easily, out of the box.

But if you add a C dependency, you also need a C cross compiler. SQLite is easy to configure, and very portable, but this still adds some friction.

Then if you want to setup CI, you need all this infra in CI. If you want to automate releases for a bunch of platforms, you need all of it in your release process.

I'm not saying it's a huge pain but it's enough that some projects avoid cgo as a matter of principle.

Many thanks for this good answer, that level of detail is exactly what I have been missing up to now.

I'm surely aware of the fact that C cross-compilation is often very clumsy implemented. But I also have to mention that there is a project that also tries to solve that too, while also creating a whole new language:

https://zig.news/kristoff/building-sqlite-with-cgo-for-every...

it's also worth noting that even on the same platform, compiling with CGO enabled creates a dynamic link against `glibc`, which creates a dependency on having the matching version in your deployment target.

We ran into this recently as our CI system is running a newer OS than we currently deploy to - and disabling CGO was an easy way to sidestep this requirement

Is this something inherent in CGO implementation? If your dependency happened because the compiled library depended on it, do you know that that is probably avoidable (I haven't tried your use case, so I could be wrong) by using zig as a compiler as it can "cross compile" for lower glibc version and also to musl, avoiding glibc altogether?

With zig I think you can avoid libc deps on Linux, at least, by statically linking musl instead.

Not sure SQLite is tested in this configuration, for the fans of “this didn't pass TH3 so we can't trust it at all.”

AFAIK: you also link to a different (lower!) glibc version from the one on which you do the compilation, probably solving the problem you had without having an excuse that you linked an "untested" library.

I’m not 100% sure why you would like to run SQLite in your go program this way (using WASM) but I think avoiding cgo in the go community is a common thing for all the reasons mentioned here [1] (or perhaps even more directly, because that article exists :-p).

—-

1: https://dave.cheney.net/2016/01/18/cgo-is-not-go

Reply on news.ycombinator.com