Multi-Kernel Drifting
luqman.ca
luqman.ca
Anyway, I guess there's a lot of practical problems in defining either a new ABI or ABI version in the ELF headers, so the different distros just sort of pretend to all be "Solaris". If I remember, Solaris (de facto?) disallows static linking, and uses dynamic linking with libc to solve the ABI compatibility problem. Wouldn't help stuff like kernel modules/FUSE, I reckon.
The biggest question to me is why Helios is using the FUSE kernel module from OmniOS. If the kernels were strictly identical, this makes sense, but if they're not, it seems weird. Since Helios is a secret Oxide thing (?), guess it's a mystery.
Makes me think of all the arcane gibberish that GNU Autotools checks for, so many minor variants in system headers and APIs...
One of the "big" guarantees Red Hat customers like RHEL for is that they will not break kernel ABI through a release cycle. They like this because it's really hard to do, and even very minor dot releases of the kernel provide no assurances that the ABI won't break.
This seems to be normal distro stuff, and the problem broadly seems to be in two parts: * The Illumos team and distros need to either figure out a better mechanism for indicating minor kernel versions than the old "patch release", or they need to actually pay attention to it now that they know things diverge sometimes. * This worked fine when building from source. The assumption that all of the IllumOS distros can freely share binary packages seems to be a bad one, and they may need to go to the same process as every other distro and actually... rebuild.
I had a similar thought but on further consideration Linux does about the same. Linux freely breaks internal ABIs at will at least across versions; you can't use modules from Fedora on Debian in practice, if nothing else then because they aren't on the same exact minor version.
> The biggest question to me is why Helios is using the FUSE kernel module from OmniOS.
It was kinda unclear, but the impression I got was that they don't in general and this was a one-off mistake where someone compiled on the wrong machine by accident.
Anyway, it was definitely just a weird mistake, I'm more intrigued about how varied the Illumos branches/distros are, since I had thought they were mostly just different userlands/package managers, when they seem to have different kernel patch sets and bugfixes...
It's not so much a secret as we've been rather busy, and I haven't had time to clean up the repositories to the extent that I'm comfortable making them public. Hopefully real soon now, and absolutely before we ship the hardware to anybody.
> different "distros" of illumos would have different kernel ABIs
We do have strong guarantees around various kernel API/ABIs, which one can absolutely use to produce out of tree modules that will work over the long term and across distributions. We call this the DDI, or Device Driver Interface, and it's documented as public and stable in our manual.
The vnode layer bits that FUSE uses are ostensibly sort of public but not documented. This is of questionable utility, to be honest, but it's apparently there. A mistake was likely made in adding support for inotify to a particular fork of illumos, and a struct member was added in the (again, wishy-washy) "public" preamble of the vnode struct.
> The biggest question to me is why Helios is using the FUSE kernel module from OmniOS
This is my fault. Before a distribution can be self-hosting, it needs to exist at all. I am a big fan of OmniOS so I had bootstrapped the first Helios bits by building modified OmniOS bits on OmniOS, and then massaging the packaging metadata until it looked how I wanted it for Helios. I then installed a Helios machine from those packages and everything was subsequently built on Helios from then on.
Regrettably I didn't realise the business with the vnode stuff, or even that we had a FUSE package. It has not been rebuilt since that original bootstrap build, so it was built with the wrong vnode.h. In my defence this is the first time in two years anybody has tried to use FUSE on Helios so nobody else noticed either!
What are the differences between OmniOS and Helios? Why did you feel the need to create a new distribution?
It's used on Debian to support out-of-tree modules: https://kernel-team.pages.debian.net/kernel-handbook/ch-vers...
The author did not disappoint at the very end of the article.
Would you automate your Linux build using VB Script? Why not?
When in Rome, do as the Romans do!
Citrix actually purchased a company like this and sold their Linux-based virtual appliance that did one thing and one thing only: create Windows disk images for running Windows applications.
It was the stupidest thing I had ever seen that was packed full of clever solutions to problems that didn’t need solving.
You realize this blog post is about Illumos, not Linux, right?
Illumos isn't nearly as well-polished.
A lot of these types of tasks are easier on Linux than they are on Windows native.
FTA: I had already gotten it working with QEMU under KVM on Linux but wanted to port it to propolis on our illumos distro, Helios.
Illumos is not Linux.
Many of the leading repair tools for Windows are Linux based, for example.
If you want to run older Windows software, you'll have an easier time on non-windows platforms via wine than on modern Windows. (Or, you can run otvdm, wine on windows itself)
etc etc
That is a completely different concept and not at all relative to building deployment images. They are not compiling Windows here on Linux.
Creating build images doesn't necessarily require anything that is Windows specific. An unattended XML file can be added to a disk image without needing to boot into a Windows environment. The challenge with Windows automation though is to get an optimized image you need WIM mounting and modification. Linux even has a WIM tool, but the tools like DISM are not really there or the Windows ADK. I'm sure DISM can be reverse-engineered and then doing automation in Linux would make a lot of sense actually since it would be less-resource intensive. Spinning up a container or Linux distro is usually magnitudes faster than a Windows machine, and Windows automation in itself sucks comparatively. I would definitely pick doing automation on Linux whenever possible as long as it gets the job done well.
No, Linux is not superior in any way for Windows install image automation!
Why the heck would you reverse engineer and port the DISM tool, when the DISM tool already exists!?.
Windows has a built-in set of free tools for automating image updates, integrations, and builds.
E.g.: you can mount a WIM image offline (without booting a VM or needing a container), and inject updates, software, configuration, or even drivers.
This is 100% supported an Will Just Work. It takes minutes, not hours.
The article is the rough equivalent of trying to build a Linux install image using Windows, via some broken FUSE EXT3 driver the causes blue-screen crashes of the entire host OS, then spending a month trying to fix that instead of just using Linux to solve the task like a normal person.
PS: I do 99.9% of my work with Windows. The one time in my career I had to inject drivers into a Linux install image, I created a Linux VM and just used that. (I had to make a version of XenServer that could boot from a MPIO FC adapter during install time, even if a path was down. Fun stuff.)
Having to configure a Windows environment for programmatic actions and subsequent changes is definitely more challenging. Even with the recent SSH access, which I believe the WinRM feature is still in preview, there are a lot of challenges for authentication that simply are not present in Linux.
> Why the heck would you reverse engineer and port the DISM tool, when the DISM tool already exists!?.
Because if you can do it in Linux, it makes sense to use an OS that is optimized to run like 95% of the world's server infrastructure due to performance and operational benefits. I really don't have time to get into an argument about why Steam OS chose Linux and why people use Wine to run Windows games on Linux. However, those are just a few indications that you're wrong.
> Windows has a built-in set of free tools for automating image updates, integrations, and builds.
They have a tool. I'm aware of Windows ADK. Everything else they had is no longer updated as far as I can tell. Most of them are just crappy GUIs wrapped around basic instructions that are as simple as creating partitions and copying files.
> E.g.: you can mount a WIM image offline (without booting a VM or needing a container), and inject updates, software, configuration, or even drivers.
It takes several minutes when in reality it is a subpar experience for developers looking to create multiple images. If someone creates a tool in Linux to do the same thing, but does it better, why does that bother you so much? Are you as bothered that Windows essentially has tried to create a Linux subsystem for Windows because developers were leaving Windows in droves because they found developing on Linux more pleasant?
> The one time in my career I had to inject drivers into a Linux install image.
You mean essentially installing a package via cloud-init which takes a second to add? Injecting drivers with Windows is still tedious if done through DISM and not the unattended XML.
No. I had to rebuild the entire image, from the boot image up, to support MPIO Fibre Channel with the capability to install successfully via PXE network boot even if one of the two fabric paths was down. This was to support large-scale deployments to diskless blade servers using a central SFTP server, where the installation target was a SAN disk array.
This is actually possible to achieve purely via Windows, using various Cygwin tools, but it is incredibly painful.
I used Linux in a virtual machine to do the build, which made this was relatively quick and easy[1] to implement. This makes sense, of course! Windows is easy to use to build Windows images, Linux is easy to use to build Linux images. Similarly, I would use MacOS to build MacOS images, and so forth.
Note that I basically never use Linux, except for rare cases such as needing to deploy the XenServer hypervisor, or some network appliance. Nonetheless, I did not "reach for Windows" as my preferred option, despite my vastly greater familiarity and skills with it compared to Linux.
That's my point. The original article author has only a hammer and sees everything as a nail, even screws. It doesn't matter if the hammer is great at driving in nails. It can be awesome at that. If you need to drive a screw, pick up a screwdriver, instead of writing clever blog posts about how hammers can be laboriously filed down to enable them to clumsily turn a screw.
[1] The requirement was complex, the actual change steps to get there were not too bad, even for a "non Linux person" like me.
I've never seen this. At least for the distros I'm familiar with it is a simple as modifying your mkinit to include a driver. Maybe XenServer is different or whatever it was that you were using.
No but Windows builds are automated with Unix tools like Perl and Make :)
> When in Rome, do as the Romans do!
This seems to be a running trend for people with a deep dislike for anything Windows-native–they tend to force non-Windows paradigms on Windows, and when this breaks, blame Windows for not being 'POSIX-compatible'.
Probably also led to the explosion of tools like Cygwin, Msys2, Mingw, etc.
Tools like cygwin, etc, exist because there (was) absolutely nothing like it on windows back in the day, or it was locked behind some paid for editor, or most often it relied on obscure hacks that were hard to read.
"find which file contains this string" is a non-windows paradigm.
Really? What about FIND and FINDSTR?
FIND command is older than Linux.