Anyway, I guess there's a lot of practical problems in defining either a new ABI or ABI version in the ELF headers, so the different distros just sort of pretend to all be "Solaris". If I remember, Solaris (de facto?) disallows static linking, and uses dynamic linking with libc to solve the ABI compatibility problem. Wouldn't help stuff like kernel modules/FUSE, I reckon.
The biggest question to me is why Helios is using the FUSE kernel module from OmniOS. If the kernels were strictly identical, this makes sense, but if they're not, it seems weird. Since Helios is a secret Oxide thing (?), guess it's a mystery.
Makes me think of all the arcane gibberish that GNU Autotools checks for, so many minor variants in system headers and APIs...
One of the "big" guarantees Red Hat customers like RHEL for is that they will not break kernel ABI through a release cycle. They like this because it's really hard to do, and even very minor dot releases of the kernel provide no assurances that the ABI won't break.
This seems to be normal distro stuff, and the problem broadly seems to be in two parts: * The Illumos team and distros need to either figure out a better mechanism for indicating minor kernel versions than the old "patch release", or they need to actually pay attention to it now that they know things diverge sometimes. * This worked fine when building from source. The assumption that all of the IllumOS distros can freely share binary packages seems to be a bad one, and they may need to go to the same process as every other distro and actually... rebuild.
I had a similar thought but on further consideration Linux does about the same. Linux freely breaks internal ABIs at will at least across versions; you can't use modules from Fedora on Debian in practice, if nothing else then because they aren't on the same exact minor version.
> The biggest question to me is why Helios is using the FUSE kernel module from OmniOS.
It was kinda unclear, but the impression I got was that they don't in general and this was a one-off mistake where someone compiled on the wrong machine by accident.
Anyway, it was definitely just a weird mistake, I'm more intrigued about how varied the Illumos branches/distros are, since I had thought they were mostly just different userlands/package managers, when they seem to have different kernel patch sets and bugfixes...
It's not so much a secret as we've been rather busy, and I haven't had time to clean up the repositories to the extent that I'm comfortable making them public. Hopefully real soon now, and absolutely before we ship the hardware to anybody.
> different "distros" of illumos would have different kernel ABIs
We do have strong guarantees around various kernel API/ABIs, which one can absolutely use to produce out of tree modules that will work over the long term and across distributions. We call this the DDI, or Device Driver Interface, and it's documented as public and stable in our manual.
The vnode layer bits that FUSE uses are ostensibly sort of public but not documented. This is of questionable utility, to be honest, but it's apparently there. A mistake was likely made in adding support for inotify to a particular fork of illumos, and a struct member was added in the (again, wishy-washy) "public" preamble of the vnode struct.
> The biggest question to me is why Helios is using the FUSE kernel module from OmniOS
This is my fault. Before a distribution can be self-hosting, it needs to exist at all. I am a big fan of OmniOS so I had bootstrapped the first Helios bits by building modified OmniOS bits on OmniOS, and then massaging the packaging metadata until it looked how I wanted it for Helios. I then installed a Helios machine from those packages and everything was subsequently built on Helios from then on.
Regrettably I didn't realise the business with the vnode stuff, or even that we had a FUSE package. It has not been rebuilt since that original bootstrap build, so it was built with the wrong vnode.h. In my defence this is the first time in two years anybody has tried to use FUSE on Helios so nobody else noticed either!
What are the differences between OmniOS and Helios? Why did you feel the need to create a new distribution?
It's used on Debian to support out-of-tree modules: https://kernel-team.pages.debian.net/kernel-handbook/ch-vers...