Having non-x86 machines makes your life harder
utcc.utoronto.ca
utcc.utoronto.ca
It's also factually incorrect:
"If you try to share things across architectures, you'll get to find all of the ways that modern systems aren't really set up for that, partly because it's been decades since most environments really tried to do this."
What? Did the author give even ONE example? No.
It has never been the case that environments haven't tried to do this. Ever hear of the Internet? It has every architecture you can imagine. If you support 32 bit and 64 bit x86, you're already "sharing things across architectures" because of how different they are.
Chris Siebenmann dates from the times when system administrators would do things like have most of the operating system executables and read-only data files remote-mounted from fileservers. We'd worry about such things as instruction set architectures and bitnessess and endinannesses of non-text file formats. We'd have "architecture-dependent" and "architecture-independent" exports. We'd take care when replacing executable scripts with compiled binaries.
And yes, it has been decades since this was a norm. DASDs have long since become big and cheap enough that just having copies of all of the files locally has long overtaken the idea of having the operating system partly on a remote fileserver that had the limited money spent on it for the really big DASDs.
Microsoft originally had DOS+Windows with a shareable "system" directory and a per-machine "windows" directory. Modern Windows doesn't even incorporate such an idea, not least because many things came along that put per-machine stuff in the "system" directory. Modern Unices and Linux-based operating systems similarly don't normalize the idea of remote-mounted /usr/share and whatnot as Unices did decades ago.
* https://utcc.utoronto.ca/~cks/space/blog/unix/FadingMultiArc...
The reality is that there have always been ways to have multiple architectures supported simultaneously, and they're neither all that complicated nor kludgy, unless one has to support poorly written software. Even back in the m68k -> SPARC transition, people had clean ways to keep things separate and clean, and to support the same homes and binaries across both architectures simultaneously.
We had the PowerPC to x86 transition show us how it can be done so cleanly that many people had no idea what architecture they were running. The same can be said about the x86 to ARM transition with Apple now. (Open)VMS explicitly supports multiple simultaneous architectures. We have ways of making fat binaries for other OSes, too - we had them back then, and we have them right now.
We also have examples of doing it wrong: Microsoft Windows. The idea that people had to know and choose whether they want a 32 bit or 64 bit OS is still incredibly stupid, as is needing to choose 32 bit or 64 bit software.
If people are writing binary files in endian specific ways, or with data sizes that depend on the specific processor, that's shitty programming. Any software doing that never had any sort of portability in mind, so that's not a good example at all. That kind of software is chided in public in the open source world, or at very least people know to isolate it and run it where it's safe.
Sharing a user's home across multiple systems, perhaps with different architectures, isn't to save disk space, by the way. Expecting users to manage data and configurations across multiple machines isn't reasonable. Shared homes is still both normal and common in larger environments.
If the author disagrees, it'd be nice to hear some actual examples of why it's a problem. After all, ARM isn't going away and RISC-V is only going to become more common.
...So why is it relevant?
Sharing files between systems with different architectures Just Works these days, unless you start getting into the really esoteric.
"Having non-x86 machines is Bad, now, today, because of problems I had with managing multiple-architecture deployments 30 years ago" is not a useful statement in any way.
Likely, the binaries are not from the OS, but from users/for users which run on different machines (and more likely than not NFS mounted across the systems to ensure consistency).
If this article was written 10 years ago, more people would probably agree. But nowadays, its perfectly possible to have a setup where all your machines, from large servers, to laptops, to small embedded devices, are x86, or aarch64, or something like that, and no mixed architecture is present. There are other, possibly more glaring, issues with sharing software across very different setups: Memory requirements, assumptions about filesystem layout, assumptions about syscalls/capabilities, assumptions about security. The architecture is usually at most a cli switch in your build pipeline, all this other stuff isn't.
That said, if you use Rust, Zig, Java, C++ (with Zig C++, for example), C (with Zig CC), building N different binaries for different targets is a bash script with exactly N lines of code. Not sure that's all that hard.
> where [...] people haven't tried their software, software has architecture specific bugs
I have to agree with this, just from the projects I've been part of, reviewed and used in the past. This is not an ARM issue, its an issue of laziness. People are inherently lazy, and they will only build and test on what is convenient. x86 machine at home, x86 ci/cd server, ... guess what isn't being tested on? Non-x86. That said, there are not that many things that can go wrong on non-x86 in my experience. They're usually little endian, they usually have 32 or 64 bit pointers, sizes, etc., and if they don't, they're likely not magical either.
People who still write C code, but also don't know that they can't just use `char`, `int` and `long` to mean 8, 32, and 64 bit, are the only population that will have issues here. Sadly, that's a lot of people in my experience.
The real issue is everything else, like I said above (fs layout, security, assumptions about OS specific behavior, ...)
https://wiki.debian.org/ArchitectureSpecificsMemo
It shows that there are no such Debian architectures.
Also floating point formats can vary a bit even with the size being the same. So a piece of code written for one might be subtly wrong for another.
And why would you not be able to treat one as an integer? Are there any examples of such ISAs?
Finally, floating point formats do differ, but that shouldn't affect anything except precision and binary format. Also I'm pretty sure "float" is standard IEEE 32 bit and "double" is standard 64 bit, with predefined exponent/mantissa sizes. The only thing that could differ between arches is endianness, and by far most ISAs have settled on little endian.
It is hardly legacy yet.
Separately, there is also the x32 ABI, although that hasn't really gained any traction.
Less of an issue if you compile from source.
Also Glibc drops support for old kernel versions time to time, so even statically linked stuff isn't safe when compiling modern stuff against really old kernels.
However, if you need more modern stuffs, then obviously you need a higher minimum version. For example, some software require SSE; or docker will only run on a modern Linux kernel.
Eg. for instance the removal of a.out support: https://lwn.net/Articles/888741/ - the a.out support was almost removed, until a single user happened to notice and object. But then someone came up with a workaround and that single (known) remaining a.out user changed his code, and the a.out support was deleted.
The interface between Userland-Userland is not, very much not, a binary compiled a decade ago, unless its statically compiled or has no dependencies, will very likely not run, and is why things such as Flatpak and Docker partially exist in the first place.
It's also why closed-source software often include every dependencies in their installation package. And even that is sometimes not enough, like when the host's glibc is too old.
They aren't so concerned about someone building it on one machine. (to be fair, they do document ways regular folks like me can do distributed builds too)
but like... there are two great pacific garbage patches, i guess its possible there are two software garbage patches bigger than the combined output software output of many countries.
And it's also just a hassle even with good build systems, and I say this as someone who has a bunch of custom build scripts to compile stuff from source.
Whether we are talking about open-source or not, there has to be a commitment to excellence at all levels of the product, as would be expected in any other real-world manufacturing.
That's pretty easy nowadays thanks to container.
You don't have to muck around with environment variables to get the build system to detect your custom toolchain anymore.
I fear that unless some kind of standardisation is introduced for SoCs, we are looking at a very fragmented future for non-x86-64 platforms.
You can easily build a kernel that will run on any PC built in the last decade and any PC that will be built in the next decade. But if you are using an ARM computer you have to run the patched kernel provided by the manufacturer.
With postmarketOS a ton of Android devices can now also boot the mainline Linux kernel. Samsung certainly isn't helping anyone run modern Linux kernels on their devices, but my old tablet released with Linux 3.1 now runs Linux 6.2.
Getting the necessary kernels and software together isn't even all that difficult most of the time, the problem is that you need someone to set up an appropriate distribution. I could've gone completely compile-free if someone had sacrificed their cloud storage space to offer compiled installer ZIPs for me, because the software on the tablet now just comes from the Alpine repositories.
> Getting the necessary kernels and software together isn't even all that difficult most of the time, the problem is that you need someone to set up an appropriate distribution
Guess what? On PC there’s a ton of people who have set up a distribution and they are all appropriate for all PCs. That makes it a whole lot easier to find such a person for your PC.
It wasn't that long ago that you needed to download five different distros to brute force a working combination if you had a laptop with an Nvidia chip.
Even today you need to pass special boot options on some AMD boards because motherboard vendors can't be bothered to release microcode updates.
Linux 5.8+ refused to boot with 12th gen Intel graphics if you had a DisplayPort to HDMI adapter installed. I believe it took at least half a year before a fix was released (in a later version of the kernel, that wasn't backported). I remember this vividly because I showed off some dumb Gnome extension and he wanted it too, so he upgraded from a lower LTS to the then-current LTS and after a reboot he could no longer boot.
My laptop didn't have sound unless I passed some kind of special parameter in the boot config somewhere. It sometimes freezes on boot while loading GDM. The display flickers and jumps around if you have a maximized application. The Nvidia driver+WM combo doesn't recognize external displays on Wayland.
I've even run into one instance where the amd64 version of the distro shipped tons of applications requiring a certain instruction set, so installing the outdated (and now abandoned) x86 versions of the distro was the only solution.
Linux Mostly Works on Most Systems. The only reason you can get anything done on it is that there are more people using it and fixing the mess that hardware vendors have left behind.
I would say that most people will be able to run Linux smoothly on their hardware if they pick a user friendly distro, but random failure modes are everywhere.
https://cdimage.ubuntu.com/daily-live/current/mantic-desktop...
Sorry I'm not sure I buy this line of reasoning.
I think that has a lot more to do with the economics around Linux, Windows, and macOS on commodity amd64 hardware becoming “good enough” to replace servers and workstations running on other, more expensive, architectures.
Not having to deal with architecture porting issues is just gravy at that point.
Sounds a little like the old Linux v Windows or the PC v Mac debate - lots of fuss and really comes down to personal preferences.
There is a big advantage to using different OSes/architectures with the same programs (compiled to target the right combination of hardware and OS) : it often helps detect subtle bugs, that may not manifest on one platform but be obvious on the next.
Especially if you have to maintain many machines (which is mostly implied in the post, but also explicitly mentioned here and there).
Hosting and maintaining a custom package repository for your self-compiled binaries so you don't have to do it on every single machine manually (maybe compiling everything from scratch might take days or the machine couldn't do it on its own anyways) is also obviously making your life harder.
My personal small scale experience, developing a small service on my laptop and running it on my local pi for testing, already ran into many of the problems. Some packages I wanted to use didn't exist, some containers I used locally weren't available for ARM. Are those unsolvable, or even hard, problems? No. But! They are time sinks that make a very simple thing more complex, more brittle and more annoying, than if I just used a homogeneous x86 setup, even if I were a skilled sysadmin (which I am not) or enjoyed fixing those differences and maintaining the fixes (which I do not).
Before you call me lazy for not patching, there are a lot of vendors that have very long patch times. If you have an environment that is mission critical, you would be well advised to not use x86 at all in the server environment. My entire income is based on cryptocurrency, so if I get owned it would result in large uninsurable losses.
So I waited a couple of years and tried again figuring they’d have got it worked out.
No dice.
Ever since then I’ve stuck with AMD64 for everything.
Running Docker containers on ARM feels like the Gentoo version of Docker. Sometimes you'll find an official image that will work, but most of the time you end up compiling images. In some cases, the base images need compiling too, so you end up with custom Docker bases that you now also need to keep track of in case they need to be updated.
There's no technical reason why it has to be this hard, but if you're mostly running other people's software, everything is just so very suboptimal.
Luckily, Apple's macbooks are now bringing consumer ARM to developers in a way that makes sense. I think there's a direct link between the M1 coming out and the sudden increase in aarch64 Docker images.
There are still major pain points with ARM but the experience is getting better.
I've stopped using it but multi arch is definitely not the reason. Biggest problem is that 60% of my punny machines resources were being used to keep Longhorn synced - and the constant high usage resource kicking the fans up started to irritate me. But the cluster worked really well regardless of what node the software was running.
When I was helping out a friend with his free oracle aarch64 server, I got baffled on how ARM feels like a second class citizen on most distros.
Also, software like Luakit has been working fine, and on videoconferencing... once you have USB 2.0 UVC webcams, FFMPEG and drivers working, FLOSS conferencing software (lots of them) will work the same.
Without architecture diversity, he wants to limit innovation, and promote a monopoly. x86 is anything but an open source standard, due to patents, and thus cannot be easily standardized. So he also voices against open standards.
My tablet and phone is ARM as well, and I'm not considering switching them to intel either.