Hacker News Reader

Comment by tylerhillery | Hacker News Reader

Oxide uses Helios, their own illumos-based distro: https://github.com/oxidecomputer/helios

They’ve written up their reasoning in this RFD: https://rfd.shared.oxide.computer/rfd/0026#_comparison_illum...

After reading that document twice, I'm still not sure why they based it on Illumos. I strongly suspect it's just down to the personal preference of the founders, which is a perfectly valid reason. This document very much reads like "here are the pieces we will use, let's work our way back to why"

The reasoning can be simplified to two things. 1. Linux does not have the bhyve hypervisor ported 2. Maintaining a Linux distribution will require more effort and have more churn than illumos.

Because Linux is just a kernel and users have to provide all of their own user space and system services there is a lot of opportunity for churn. Illumos is a traditional operating system that goes from the kernel to the systemd layer. Illumos is also very stable at this point so most of the churn is managed up front

The choice is between porting a handful of apps to illumos or jumping on to the Debian treadmill while pioneering a new to Linux hypervisor. Would Linux have enabled a faster development cycle or just a easier MVP?

There's no churn in a graveyard, either. Debian's not much of a treadmill on stable; it's famous for it.

The justifications for bhyve over kvm are similarly inscrutable; you can simply not build the code you don't want. Nobody's forcing you to use shadow paging. Comments like "reportedly iffy on AMD" are bizarre. What does "iffy" mean? This wasn't worth testing? Why should I, a potential customer, believe that these people are better at this than the established companies who have been producing nearly-identical products for twenty years? At the domain of development they're discussing why bother using an x86_64 processor from a manufacturer who does not bother to push code into the kernel you've chosen?

Again, it's their company, and if they (as I suspect) chose these tools because they're familiar, that's a totally supportable position. I just can't understand why we get handwaving and assurances instead of any meat.

You may disagree with our rationale, but it is absolutely absurd to complain that that RFD 26[0] does not have "any meat." This is in fact dense technical content (10,000+ words!), for which I would expect a thorough read to take on the order of an hour. Not that I think you read it thoroughly: you skimmed to parts, perhaps -- but certainly glossed over aspects that are assuredly not your domain of expertise (or, to be fair, of interest to you): postmortem debuggability, service management, fault management, etc. These things don't matter to you, but they matter to us quite a bit -- and they are absolutely meaty topics.

Now, in your defense, an update on RFD 26 is likely merited: the document itself is five years old, and in the interim we built the whole thing, shipped to customers, are supporting it, etc. In short, we have learned a lot and it merits elucidating it. Of course, given the non-attention you gave to the document, it's unlikely you would read any update either, so let me give you the tl;dr: in addition to the motivation outlined in RFD 26, there are quite a few reasons -- meaty ones! -- that we didn't anticipate that give us even greater resolve in the decision that we made.

[0] https://rfd.shared.oxide.computer/rfd/0026

I did indeed read your document (twice, as I explicitly reported). I didn't address those parts because I found them better-supported. Instead, I addressed the parts I found confusing, and since your rebuttal here is just whining about what you think my behavior is, I continue to be mystified. That's okay; nobody expects you to explain yourself to me. If I thought it would help, I would suggest that perhaps a more effective defense would involve answering literally any of the questions I already asked. However, I don't appreciate accusations of bad faith based on your unwarranted assumptions about what I did or did not do and, bizarrely, what you imagine my motivations are. I'll just assume that the answers to the "why" questions I asked are rooted in similar wild-ass speculation.

...but your top post didn't ask any questions; certainly not ones that would justify a detailed answer.

It was several assertions, plus your admission of confusion. I mean, there are no stupid questions, but there wasn't even a question there, so I don't blame anyone for thinking you're communicating poorly.

I wasn't accused of communicating poorly. I was accused of lying about reading a text file and having some kind of ulterior motive for my own opinions.

Furthermore, advanced readers are generally able to infer from "I am not sure why x" that a similar flow of discussion might be as feasible as if it were phrased "why x?".

As though that were necessary. There's plenty of room in this comment box for questions better fleshed out than "why x?". Are you expecting advanced readers, or clairvoyant ones?

There is a reasonable explanation for the "foregone conclusion" flavour of the RFD that doesn't cast aspersions (quite as much as you are) on the authors:

It is simultaneously an assertion of the culturally determined preferences of a group of people steeped in Sun Microsystems engineering culture (and Joyent trauma?), and a clinical assessment of the technology. The key is that technology options are evaluated against values of that culture (hence the outcome seems predictable).

For example, if you value safety over performance, you'll prioritise the safety of the DTrace interpreter over "performance at all costs" JIT of eBPF. This and many other value judgements form the "meat" of the document.

The ultimate judge is the market. Does open firmware written in Rust result in higher CSAT? This is one of the many bets Oxide is making.

Frankly, I don't think Oxide would capture so much interest among technical folks if it was just the combination of bcantrill fandom + radically open engineering. The constant stream of non-conformist/NIH technology bets is why everyone is gripping their popcorn. I get to shout "Ooooooh, nooo! Tofino is a mistake!" into my podcast app, while I'm feeding the dog, and that makes my life just a little bit richer.

i'm not sure if Dtrace interpreter was safer than EBPF. I guess in theory it should be because a JIT is just extra surface area but I'm not sure in practice. Both EBPF and DTrace had bugs. Also, I always thought EBPF JIT was just a translation to machine code and it didn't do any kind of optimization pass so should be very similar to how DTrace works. They both ship byte code to the kernel. But I guess the big difference is EBPF relies more on a verification pass while I think most of DTrace safety verification was performed while executing the bytecode. I remember there was a lot of stuff in EBPF where the verifier was meant to be able statically determine you were only accessing memory you were able to. I think there was a lot of bugs around this because the verifier would assume slightly different behaviour than what the runtime was producing. But this is also not necessarily a JIT problem you could have an interpreter that relied on a static safety pass as well.

Reply on news.ycombinator.com