Spectrum OS: a declarative, reproducible, compartmentalized Linux
spectrum-os.org
spectrum-os.org
Nix seems cool, but all of my forays into it have been so unpleasant as to be unworkable. It seems to work well enough if all of your dependencies are already in the Nix store and have been thoroughly tested, but as soon as one has to start writing Nix packages themselves it's a train wreck: the peculiarities of the Nix expression language, the chaos and lack of documentation in nixpkgs, the immense understanding one must have of the common low-level libraries used to create Nix packages, and you'll still spend tons of time trying to package obscure C dependencies with their own bespoke build systems.
My constructive feedback is:
1. Add types to the Nix expression language so someone digging through the code can have some idea about what needs to be passed into various functions. This would probably help people traverse nixpkgs as well, since the client code will need to "import" the types of its arguments.
2. Make Nix more syntactically familiar. Familiarity here seems like it should be more important than innovating on programming language syntax. Thanks for not going full-Haskell on us, but it would be nice if looked more like JavaScript or Python or something that virtually any programmer could look at and recognize (I'm no great fan of either of those languages).
3. Very controversial, but the whole industry needs to minimize the number of C and C++ dependencies. Not only are these languages fundamentally insecure, but projects in these languages have their own bespoke build systems which assume dependencies are already installed at the correct versions and in the correct paths. Packaging these projects is painful, and it's largely the reason we have package maintainers who specialize not in building programs of a certain language, but in building certain dependencies.
It would have been so much better if they went full Haskell. Haskell is better designed as a language, it has types and could support all Nix features without modification as far as I can tell. On top of that it's a common language that most have used at least in university and has loads of friendly tutorials online.
Totally agree on the packaging of C/C++ projects is a drag of our entire industry and basically a complete embarrassment.
One simple thing I like about Haskell is its readability. Instead of cramming both type signature and function definition into the same line, Haskell splits those two into separate lines. Having the type signature in its own line, as the first line of every function, is so much more readable.
It's also interesting that you can outline/pseudo-code an entire program structure using only type signatures and no function definitions, check if the type signatures are all correct, and then fill in the function definitions later.
That said, I'm still new to Nix, and there are probably good reasons it was done this way, buried in past discussions, the creator's PhD thesis, etc. that I haven't read yet. And I hear that types are being looked into for future upgrades to Nix.
Is this bogus, or would you imagine that a substantial number of additional wrapper/helper/noise functions would be needed to make a general-purpose language do what the Nix language does?
I guess Guix is probably an opportunity to look at a practical instance of this first-hand.
The ability to use file paths in Nix gives you a very cheap mechanism for splitting code across files. In Nix, the expression `import <filepath>` will evaluate to the result of evaluating the file `<filepath>`. But in Haskell, every such import would need to be declared at the top of the file, and the module name would need to be duplicated at the use-site, almost certainly fully qualified to avoid nameclashes.
I'm not overly bothered by type systems for things like Nix. In Nix, you're mostly just coding the construction of a single, reproducible value, which is trivial to exhaustively test: just construct it and see if it's right. If there's something wrong, you'll either
1) get the error immediately when you try to evaluate, and you might even be told helpful things like a list of strings was expected, not a list of numbers.
2) get an error when the derivation is built, due to an environment error that is unlikely to be detected by a type system.
Splitting this into two phases, compile then evaluate, wouldn't help me much.
I think the main costs of Nix being its own thing are probably in areas like documentation and performance. But a lot of the documentation issues with Nix aren't actually issues with the documentation of the language (which is fairly minimal) but rather with the document of nixpkgs, which contains mountains of magical helpers and other tricks with horrendously bad documentation and discoverability.
That infrastructure (and the corresponding documentation gap) would likely exist regardless of the base language.
Also, I think your claim that "most have used at least in university" is skewed to your experiences. I doubt the majority of programmers have any experience in Haskell whatsoever with even fewer having enough experience to be comfortable in it.
It depends on what, precisely, you are measuring the difficulty of. Are you writing a quick script? A complex software project? Trying to master the language? Is it a beginner programmer? A narrowly focused coder who only knows one or two similar languages? An experienced programmer with a wide breadth of knowledge?
In pedagogical terms, I found learning Haskell much friendlier when I was contemporaneously learning Java, C++, Prolog, etc. in university.
One of the big reasons that Haskell is hard to learn is because it is purely functional. Writing idiomatic Haskell is dramatically different from traditional imperative C-like languages.
Just to give an example - a few years ago I was taking an AI class which used Common Lisp (a functional language, but not purely functional). The first few tests/projects which required functional lisp programs led many of my peers to drop the class.
I think part of this is because my school, like most, started students off with an imperative/procedural language (C++). If you have a clean slate (e.g. someone with zero programming knowledge) it’s probably just as easy to teach them Haskell as it is C++ or Java.
Functional programming is great, and I love it, but many programmers don’t even have a firm grasp on recursion which makes pure languages like Haskell like a mountain to climb.
It's definitely not.
Source: I know a lot of people who had Caml classes before C classes, C was much easier. Also in my school we learned C and LISP at the same time, most people also found C easier.
I suspect that if someone has a strong background in mathematics they might take to functional programming easier than they might other languages. I could definitely be wrong though!
> many programmers don’t even have a firm grasp on recursion
I can’t say I ran into more than a few people like this and I don’t think any of them stayed in CS.
There are plenty of programmers out there that learned at boot camps, universities with poor CS programs, or who are self-taught, who likely have a much less solid understanding of CS fundamentals. Many of those programmers have never needed to understand recursion. I went to a school with a very average CS program, and many of my peers struggled with recursion. There were more than a few who passed the class (Data Structures) and graduated alongside me despite not fully grasping the concept of recursion.
Some of the best (functional) programmers I know learned (functional programming and programming in general) at boot camps. I think the predictor for FP proficiency is not education (and I hesitate to share my predicted predictors here). In any case, they certainly understand recursion.
That’s showing a bit of bias.
Personally Ive been in the industry for roughly 20 years and I’ve yet to meet anyone IRL who has even dabbled in Haskell.
Basing your packaging-format on Haskell is a great way to alienate most Linux-users without a specialised university-education, and I’m wagering that’s a fairly big portion of desktop-Linux end-users.
See that's the bubble ;) If you work at a company who makes archival, language-detection and ocr you will see C/C++ Haskell and OCaml..well and perl in no time.
I think your scenario sounds a lot more like a bubble. It's certainly my experience that very few people have encountered Haskell whether in a university setting or a professional setting. Yes, there are some niches where Haskell is "not rare" (especially the programming language theory corner of computer science departments), but in general I think it's quite rare.
Depends on the job you have, and how far you go out of your bubble.
>but in general I think it's quite rare.
Like Fortran...but not if you work at Cern.
And Cobol...but not if you work at a Finance institute.
And PL/1...but not if you work for IBM.
Guix is essentially a reimplementation of nix packaging system using a pre-existing language - guile scheme.
Some may react adversely to them using a lisp (<s>well, enlightenment is not for all ... </s>), but leaving that aside, guix has continued to evolve in slightly different ways to nix, but still the core (and the guarantees it provides) remains essentially the same
It's been amazing for me. Especially when combined with home-manager [1], which provides declarative user environments and flakes [2], which provide a even more declarative and more easily shareable package format.
Getting my exact setup on a new device - including lots of GUI and terminal app customization - takes a single `nixos-rebuild-switch`, and works every time. Everything is configured in a single config file, from hardware and system service setup, over window manager setup, all the way to installed Vim/VS Code/Chrome plugins and lots of application configs.
The ability to easily roll back to previous configurations and boot into old configs from the boot menu in case something breaks is also brilliant.
You can also just install and run pretty much all software on demand, similar to `npx`, without polluting the system. ( nix run some-app). Declarative, reproducible development environments for each project are the cherry on top.
That said, the onboarding experience is horrible. There are A LOT of things to learn, and the documentation is bad. I also agree that the language, while somewhat fine, is not great and very undiscoverable.
The Nix ecosystem is a diamond in the rough. Sadly it would take a lot of effort to simplify everything and make it more polished.
I'm afraid Nix will continue to remain very niche.
[1] https://github.com/nix-community/home-manager [2] https://nixos.wiki/wiki/Flakes
> Sadly it would take a lot of effort to simplify everything and make it more polished.
Yes, and while it would take a lot of effort to make most things nicer (improving the language, etc), other things require effort that scales linearly with the number of packages in the universe. For example, making the package definitions in the first place, adding documentation for the packages, testing packages, etc. Other things are even worse: testing various combinations of dependencies at different versions.
> I'm afraid Nix will continue to remain very niche.
Unless it gets some serious corporate support, I'm afraid I'll agree. The Nix folks have done some impressive work, but I get the impression that they're more interested in making something cool for themselves and other like-minded people at the expense of others (and there's nothing wrong with that! it's just incompatible with growing out of a niche).
Cmake can be worse, Meson versionitis can be annoying. It is also annoying to install Meson and Python in stage 1 in Linuxfromscratch now. Yuck!
The languages aren't fundamentally insecure, there are tons of reliable projects out there.
I mean I built a whole nix distribution for embedded devices with runit as init and cross compiling and everything and nix made it easy and fun. And I learned that it's just Autoconf and shell all the way down. There's no magic to nix. The most voodoo non standard thing they do is patchelf stuff, but most of the nixos specific stuff is just setting paths to configure scripts or in the environment.
I don't really understand your line of argument.
I'm in the process of moving my website over from AWS to my own infrastructure (I was using S3 and their CDN for my blog, but unfortunately, the Parler debacle made me rethink my association with them), but the repositories (which are old and need to be updated... this is my hobby project) are on github.
This is my nixos extension to use musl and runit instead of glibc and systemd: https://github.com/intrustd/appliance
built off of my own fork of nixpkgs: https://github.com/intrustd/nixpkgs (forked from a the 19.* releases I think... some changes need to be merged upstream).
Currently, I have it running on an O-Droid HC2 and have used it to share my photos with family. Unfortunately, most home internet connections just don't have enough bandwidth even for modest photo and video sharing, so I'm exploring other means of data transfer rather than a strict client/server thing (maybe using BitTorrent over WebRTC to share larger files). In either case, all the data remains owned by my and physically with me. The system can generate URLs that can be used to access the device for limited amounts of time. So, for example, I can ask my photos app to generate URL I can send to my aunts to show them pictures of my daughter, and this URL will last for X many days until the token expires.
I know this is a non-issue for your case but I wonder how much more expensive would self-hosting a blog (+ a few web-services) on a SBC be w.r.t cloud solutions? Have you calculated the numbers (electricity usage, traffic, etc).
My Odroid HC2 takes very little electricity. Haven't run the numbers on it.
So yes, all this has been... somewhat harrowing. But overall I would say I'm quite pleased with the end result and impressed with what Nix has enabled. Despite its long evolutionary growth, many of the things that I've encountered feel thoughtfully considered and well made. The various ways that builds and binaries can be pushed around and shared is awesome, and I'm excited for the optimizations coming with the content-addressed store.
I don't think that making the Nix language or package definitions feel more conventional would necessarily be a good thing— I think there's a risk that those kinds of changes obscure the underlying realities and lead to expectation mismatches.
That said, one thing that I do feel is pretty unfortunate is the interplay between the shell and Nix in package definitions. Basically you pass a dict to mkDerivation, and all the stringy items in that dict become available to your build's environment as envvars. This means when writing out strings for your configurePhase, buildPhase, etc, you have to be hyper-aware that ${thing} will be templated by Nix in the Nix context, whereas $thing will be templated much later, by Bash. It's something that is learned quickly but my goodness is it ever a hell of a tripwire.
> passAsFile: A list of names of attributes that should be passed via files rather than environment variables.
Looks like it's basically just a hack for envvars that do/might exceed whatever bash's internal limits are for them.
Interestingly, this was one of the first issues that Edolstra (creator of Nix) raised himself, but unfortunately he closed it again as unrealistic a number of years later [0].
[0] https://github.com/NixOS/nix/issues/14#issuecomment-37820190...
See esp.: https://github.com/tweag/nickel/#related-projects-and-inspir...
I don't think we should just ignore the immense technological knowledge encoded in C/C++ projects throughout the decades, only because they came from an era which having a package manager for every language was not fad of the day.
Let's see how those "memory safe" languages would fare 20 years down the line...but I predict there would be a cool_lang and people would advocate for rewriting every algorithm known to man, yet again
Should the default "webserver" really be written in c? Or maybe just the one for embedded or high performance?
> Let's see how those "memory safe" languages would fare 20 years down the line...but I predict there would be a cool_lang and people would advocate for rewriting every algorithm known to man, yet again
This just reads like you've taken personal offense. My argument is quite clearly not "C isn't cool enough" or "C isn't memory safe", it's "C lacks a standard, sane build system such that packaging C projects is a nightmare".
I mean, yeah, right now, Rust/Go have much better tooling than C/C++ for packaging and dependency resolution it's not even a debate but still one may argue that C/C++ library management is decentralized :) and maybe in future people converge to some unique method of dependency resolution that would deprecate Rust/Go way of doing things (like Nix/GuiX).
I came to this view because I see lots of cool projects in very specialized fields, that are practically abandoned and no one continues working on it on the sole reason of community thinking its not worth continue maintaining it in old lang/tech.
Again, consider I'm mainly a embedded guy...so idk, maybe a fresh re-write of every classical cs problem every 10 years is better in practice...
I'm not saying that we should deprecate C because it's old or unsexy. I'm saying there are practical problems with the C ecosystem that are unlikely to work themselves out which makes packaging downstream software really, really hard. This is a bad fit for domains (like SaaS) where the pace of software development is quite rapid.
> so idk, maybe a fresh re-write of every classical cs problem every 10 years is better in practice
In this case we wouldn't even need to rewrite things if the maintainers of these software projects would opt into a sane package manager (there exist sane package managers for C, for example, Conan) but most maintainers of C projects are militantly opposed.
https://www.cvedetails.com/product/47/Linux-Linux-Kernel.htm...
but my question would be: is this exactly the same issue that all other package maintainers have e.g. rpm, deb etc?
https://alyssa.is/leaving-freeagent/
“When I had a call with [the project sponsor] NLnet recently, I was told ‘you work for the internet now’. And that evoked the most powerful feeling of purpose and drive I’ve ever felt in my life.”
“Although my funding is for Spectrum, I expect to be able to work on free software much more broadly over the next year. Certainly, a lot of my work will make its way into Nixpkgs, which I was very excited to receive commit access to some months ago due to my other work on it. And there are some further projects of my own to come as well. In addition to working on Spectrum, I’d like to diversify my funding. I hope to be able to take donations for my free software work on Spectrum and other projects, to make this important work truly sustainable for me. I couldn’t sensibly do this before, because I couldn’t guarantee that I’d have any time to spend working on what people would be giving me money for, but, in this new arrangement, I have a lot more freedom and capacity to choose how I spend my time. I don’t expect donation income to pay my bills, or even be significant, but I’d like to explore it, and until now I haven’t had the space to do that.”
I've moved all my machines to NixOS, and even replaced Homebrew with Nix on my work OSX laptop(with home-manager). It's a rather refreshing system. Being able to use nix-shell and temporarily install one-off packages is also great.
The motivation behind this system sounds great.
If this solves that problem i'd definitely be interested in trying it out. As the only thought i had was manage to put 100% of my state into my nix config which seems.. a nightmare.
Have you tried setups like "Erase Your Darlings" and "NixOS: tmpfs as root" to explicitly persist and manage state?
Big PITA though, imo. At least until i'm "better" at Nix.
[1]: Some things, notably containers like LXC/LXD, can’t work with tmpfs mounted to /var, /var has to be on disk, while the rest of / can be tmpfs.
The problem seems to be how to do it securely. You could imagine faking GPU hardware for the VM image to talk to. But maybe Vulkan would be a better API? Has any work been done on a wrapper forwarding calls that refer to VM-mapped addresses to a native Vulkan? (What happens in WSL2?)
I find Kompute ( https://github.com/KomputeProject/kompute , https://www.youtube.com/watch?v=Xz4fiQNmGSA ) that seems to be a serious attempt at building general computation infrastructure on top of Vulkan. Is exposing a Vulkan wrapper, and building compute kernels on top of that, a practical avenue?
This all seems like something someone must have throught through already, but I have not found anything published about it.
You can do this if you have a second GPU: https://www.qubes-os.org/faq/#can-i-run-applications-like-ga...
https://forum.qubes-os.org/t/qubes-gpu-passthrough/661
https://forum.qubes-os.org/t/another-2-gpu-passthrough-post/...
When Apple started, they were sued by Apple Records, the Beatles' label, and settled with a pledge to never have any activity involving music distribution.
Come iPod, they turned around and sued Apple Records.
in reality, maybe something like this that builds on existing primitives to get there may be the only real, feasible thing... but assuming you could start from scratch, what would it look like?
> getting away from memory unsafe languages
How about secure virtualization and security through isolation instead? See Qubes OS.
yeah, that's kinda what i meant by "something like this that builds on existing primitives to get there may be the only real, feasible thing".
it looks like qubes uses vms as an existing (heavyweight) primitive to limit blast radii for existing software and systems. i wonder if you can do better if you start from scratch.
It's not exactly heavyweight, because Xen hardware virtualization eats up only additional 10% of CPU. However, GPU acceleration is switched off for security reasons.
https://github.com/NixOS/nix/issues?q=is%3Aopen+is%3Aissue+m...
I expect the delay is mostly due to all this debate raging about `nix shell` naming and semantics: https://github.com/NixOS/nix/issues/4715
I give to QubesOS every month, already. QubesOS uses OpenCollective to administer donations, which seems to work.
Seems like that could better be linked from somewhere easier to find...
I'm going to give it a try.
Neat!
I still use djb's ucspi all the time.
(stunnel or haproxy to add TLS support)
In fact, I will predict right now the death of Linux as a dominant backend computing platform. It's sort of happened already. Sure, Linux is the kernel that runs the host machines that the cloud runs on. But the services are increasingly serverless/kernel-less, or run on micro-vms running either Linux or another kernel, or run in containers made of a half dozen different base distros. All the Linux I/O is just passed to an outer layer with a universal adapter, so you can mix and match networking, filesystems, logging platforms, policy governance, orchestration, scheduling, etc. I spend nearly all of my time building systems by tuning software and services that have almost nothing to do with Linux.
Linux distros will still be around in one form or another. But long gone are the days where most people define their system or security based on a particular distro. Hell, most of the software used in the cloud today isn't even packaged by distros. Linux has become just a kernel again, the distros just a fancy installer for GNU tools.
They're using a lot of advanced tech and advertising having to "maintain fewer virtual computers", whatever that means. But it also seems their tech is more complex and less compatible with existing components. There's no clear idea of how a developer's supposed to take their code on their laptop, test it, and ship it to production as one immutable image. If it's not as simple as containers, it's not going to replace them.
Also, it's interesting that they seem to allude to immutability, but don't mention the principle by name at all. It seems like whomever is developing this doesn't run large distributed systems.
Since this system is based on NixOS and the Nix package manager, immutable is implied. Any change you make to a Nix/NixOS system results in a full rebuild of the system, and pointers to the new system updated. Current and all prior builds remain immutable.
Keep in mind that all large-scale systems work based on deploying container images. If it's not in a container image, it's not getting deployed.
It's like the difference between shipping an app and shipping an OS with the app-preinstalled, except in this case neither has any external dependencies to worry about.
(Dockerfiles are not the only way to build images, of course, but they are a common one).
Installing packages is only a small part of building a typical container image, especially for custom apps being built to run in the cloud. Even if you have your dependencies, you need to compile, run tests, and potentially customize the files inside the container, typically using the programs already installed in the container.
So since you're already running a pipeline of scripts in a container right after installing package deps, I'm not sure why one would use Nix for this when Debian works just as well?
I think the main advantage is the builds are actually deterministic. Any Dockerfile with the equivalent of
apt-get update
So say that you need to make a Docker container running nginx hosting a static website that you build with Jekyll. At a high level you'd write a Nix file like (very much pseudocode):
with let site = buildWithJekyll src={"./src";}; let mynginx = nginx { wwwroot = site; }; docker.buildLayeredImage { inputs: mynginx, entrypoint: "./nginx" }
Nix will run Jekyll to build the site, hash the build output and "site" gets that path. Then Nix takes Jekyll and writes a configuration file that sets wwwroot to the Jekyll output. Then Nix builds a container that runs nginx when it starts. If you just change the Nginx config it won't rerun Jekyll because it knows Jekyll's build inputs haven't changed.
Nix is a build system that runs other build systems. Comparing Nix to Debian doesn't really make sense, it's like comparing build pipelines to makefiles.
But I believe NixOPs is the canonical way to do what you're describing in production/at scale:
https://github.com/NixOS/nixops
https://nixos.org/nixops/manual/
(more below)
If you want a more experienced answer, I suggest asking on the NixOS forum or subreddit, people are quick to answer in both places.
https://www.reddit.com/r/NixOS/
----
More useful NixOPs guides
https://www.thedroneely.com/posts/nixops-towards-the-final-f...
Our industry shamelessly imitates of the limping, stagnant, service economy in doing these things, and we call ourselves the future!
Nix is more (or less) than a container. It depends, what you are targeting with your nix-project. You can either build a minimal NixOS, that has only a kernel, the minimum required user land and your project and its dependencies.
Or build a stand alone docker (etc.) image and spin it up.
Or you can use the nix shell to replicate just the dependencies of your project in that sub shell for other developers or testing.
I just recently saw a convincing video about nix, comparing it to docker.
Whoever said that, such wise words.
Never thought of it this way, but it truly reflects how it feels to work on open-source projects; no national, religious or corporate boundaries to honor is just liberating!
Qubes-Lite with KVM and Wayland (roscidus.com)
141 points by todsacerdoti 6 months ago | 48 comments
I think they are completely different from NixOS. For starters, they want to run every app inside crosvm.
> Spectrum will, for now, be a Linux-based system, derived from NixOS.