Hooking Go from Rust
metalbear.co
metalbear.co
In case the author happens to see this and can respond, how did you figure all of this out? Would really enjoy a deep dive covering your process. This is brilliant.
I also wonder if it could be extended to Nim, Zig, or any other less-straightforward hookable languages [than C/C++].
Edit: Apologies, maybe this is not that interesting of a wonderment. I did a little research and both Nim and Zig interface with libc.
I'm not yet clear on whether node.js or Deno use libc, if any fellow HNers know please leave a reply! If they don't use libc, they could be interesting targets.
regarding extending, probably possible - I think it'd be easier with Zig (no weird stacks AFAIK) but we don't need it as they probably just use libc like most languages.
P.S would love hearing your thoughts about mirrord
Zig does not depend on libc on Linux, for example: https://github.com/ziglang/zig/blob/master/lib/std/os/linux....
You can choose to link libc though.
Could someone explain this. I'm not familiar with low level linux stuff. Why would you choose not to use libc, what are the implications?
1. libc is considered hazard - security, comfort, runtime. It needs to maintain support for so many flows and setups so it's hard to make things better. For example, having `errno` as a global is quite weird (instead of just returning it, which is a whole discussion of it's own)
2. Golang devs like to re-invent the wheel, in a kind of Apple-ish way - all other are doing it wrong, we're going to do it better. ofc it's debatable whether they're correct with their approach.
3. Given they use Plan 9 system design, doing FFI from Go is very expensive (need to switch stack, save context, etc on each call)
It was quite weird before we wrote concurrent software. Once you're writing concurrent software it's very weird because as originally documented this single error location is shared by all threads, so it necessarily gets concurrently modified while being read, a Data Race.
Your 2022 C library typically defines it as Thread Local instead, so when you call some_c_function() and then check errno, you aren't confused by the fact meanwhile another thread tried to open("nonexistent.file") and so their thread set errno to report that the file doesn't exist.
Given that none of this fuss is how the actual system calls work, it's understandable that people might want to sidestep it.
I'm very happy with these tradeoffs. Not only is it a pleasure not to have to deal with libc (truly static binaries on Linux, 2mb docker images, etc), but because interop is tedious/expensive, the ecosystem is largely pure-Go. This means that we rarely (if ever) have to deal with C's abysmal build tooling and (lack of) package managers and stuff like cross compilation is trivial. Everything builds consistently on every system, every time, and irrespective of target platform (provided there are no C dependencies).
Easy interop has only ever been an illusion (we forget about the difficulties of integrating distinct build systems and memory management systems or the runtime costs of marshaling data between memory layouts). Rather than paying significant costs to preserve that illusion, Go doubled down in order to realize some pretty considerable gains: easy compilation even on niche distros, even when cross compiling, truly static binaries, etc.
I don't dispute this, but that is still a niche case. The general case is that library software gets rewritten in each language (and if it's not worth the trouble to rewrite it, it's usually not worth the trouble to use and maintain bindings). So yes, there is a tradeoff, I'm just expressing my opinion that Go made the better tradeoff, at least for a garbage collected language (since you mentioned it, Python trades off a boatload of performance and package management simplicity in its pursuit of easy C interop).
> Yes C lacks dev ergonomics, but meson + ninja gave a very neat experience..
My point about abysmal C build/package tooling wasn't about the experience of a C developer, but the experience of someone who is downstream of C packages. As a Python developer, I don't get to choose how my C dependencies are packaged for Python (which build system and package manager they use)--I'm just at the mercy of whatever they offer, and if I'm targeting some non-mainstream Linux distro, then there's a good chance some build or runtime dependency of some upstream C project is going to fail, and now I need to grok that C project's build/linking system to sort it out. This just doesn't happen in (pure) Go because Go's build system and package manager are reproducible (no implicit dependencies).
> "Goroutine stack is dynamic, i.e. it is constantly expanding/shrinking depending on the current needs. This means any common code that runs in system stack assumes it can grow as it wishes (until it exceeds max stack size) while actually, it can’t unless using Go APIs for expanding. Our Rust code isn’t aware of it, so it uses parts of the stack that aren’t actually usable and causes stack overflow."
tl;dr - yes, you're correct and we replace Go stack with system stack for the duration of the call, just like cgo does.
Does anyone have any recommendations on how to do something like this but in reverse? Calling a go function from rust?
Do you mean for an already compiled Go binary? if you can recompile you can use cgo.
If not the following requirements come to mind (there are more probably):
1. Allocate g and m if doesn’t exist in current process. 2. Switch to go stack before call
Btw there’s the cgocallback routine that manages C code that calls into go so that’s a good look to see what’s needed.
The approach in this blog works with compiled binaries.
also found a discussion - https://news.ycombinator.com/item?id=18439100 that points to why this might be the case