Hacker News Reader

Comment by zdkaster | Hacker News Reader

It's a private cloud. I guess you might not be a target customer.

I develop cloud infrastructure controller software for a living but sure. I’m certainly not a target customer, but as someone casually interested in this space it’s certainly very light on details.

Re-assessing my criticism, I think I’m mostly complaining that I can’t figure out what software it’s running. But I guess the target customers don’t care, since it’s an implementation detail. I’ve only ever heard “oxide” in the past in reference to the OS they were (are?) developing, which a web search says is called “hubris”, so when I see an article like OP I’m wondering “are they running k8s on top of hubris? Wow!” But I’m imagining that’s very much not the case.

In the comments in this very section an oxide employee mentions they’re using Illumos, but I don’t see that anywhere on their website either.

But you’re probably right in that the target customer doesn’t give a shit what OS it runs, so long as they can get instances deployed (although I would question why, if you’re going to run k8s anyway, you don’t just run it on bare metal and skip the hypervisor, but that’s my bias showing up as someone who writes bare metal controller software.)

You're completely skipping the management of the bare-metal hardware itself. Sure, you can buy a stack of whatever commodity hardware you want and run Kubernetes on bare-metal. What about updating the firmware on that hardware? What about expanding and the hardware you purchases is longer available? Now you have more disparate hardware that may not even be compatible with the software you're running. Those problems exist at scale and is what Oxide is tackling. It's more than just buying a computer and set it and forget it.

I'm not really criticizing oxide as a product here, I'm sure it's great. I'm criticizing the website for being heavy on "solutions" and light on details. But really, the main detail it's lacking is the operating system it uses, so as I said, I needed to refocus the criticism.

But to your point, I don't think I ever said machines don't need a management control plane (I develop a bare metal management control plane for $DAYJOB, I'm fully aware of what it entails!) I'm saying that you don't need to insert a VM layer between the bare metal and containers: Having k8s run on the bare metal OS would be ideal IMO.

But since Oxide seems to be Illumos-based, that's basically not possible... At least not for containers as most people know them (ie. with linux-based docker images.) Heck, even if running a VM layer between the OS and the containers, it looks like running your own hypervisor prevents you from (currently) having nested virtualization, which makes certain k8s workloads a lot harder.

"Light on details" is not often a criticism that we get, but I think you might be looking in the wrong place: for the kind of detail that you're looking for, you should be heading into our RFDs -- specifically (for the issue you seem to be most interested in), RFD 26.[0][1]

[0] https://rfd.shared.oxide.computer/rfd/0026

[1] https://news.ycombinator.com/item?id=41515447

I said some of this elsewhere, but the operating system is an implementation detail since we're not selling you bare-metal compute--we're selling you on-premises cloud computing. I think you were looking at this from hardware plus operating system rather than hardware plus a cloud computing control plane.

I get what you're saying though, and you're right that certain workloads like Kubernetes on the metal that can take advantage of hardware wouldn't be a good fit for Oxide today. We could decide to change that in the future, but I don't see that as a priority for Oxide given all else we have to build first.

Its funny because Bryan has spend a good part of the last decades walking around and telling people that they should have VM layer. And this criticism can credibly made against his company.

Interestingly he has mentioned that in the podcast and basically said, the industry has standardized on VM and in the kind of environments they run, there just isn't a good way around it.

But yeah, a world where Illumos based bare-metal containers had become the standard would be a nicer. But its not where the market is.

(To answer one of your specific questions, "hubris" is an embedded RTOS and is what runs the BMC-like service processor, and illumos is the host operating system.)

Reply on news.ycombinator.com