Also, it has little to do with the cloud; it is yet another hyperconverged infra.
Weirdly, it is attached to something very few people want: Solaris. This relates to the people behind it who still can't figure out why Linux won and Solaris didn't.
Yes they are using illumos/Solaris to host this but they don't sell on that, they sell on the functionality of this layer — allowing people to deploy to owned infra in a way that is similar to how they'd deploy to AWS or Azure. How much do you ever think about the system hosting your VM on those clouds? You think about your VMs, the API or web interface to deploy and configure, but not the host OS. With Oxide racks the customers are not maintaining the illumos substrate (as long as Oxide is around).
You could be right about demand, there is risk in a venture like this. But presumably the team thought about this - I think folks who worked at Sun, Oracle, Joyent, and Samsung and made SmartOS probably developed a decent sense of market demand, enough to make a convincing case to their funders.
Now you need to know both the OS they chose and the OS you chose...
(No, I don't believe it'll be 100% hands-off for the host. This is an early stage product, with a lot of custom parts, their own distributed block storage, hypervisor, and so on.)
I wouldn't have picked Opensolaris, but it's a lot better than other vendors that are either fully closed source, or thin proprietary wrappers over Linux with spotty coverage and you're not allowed to touch the underlying OS for risk of disrupting the managed product.
GNU/Linux was also "risky" at some point.
This appears to be a much better system than VMware, is free as in software, and it builds upon a free software operating system with lineage that predates Linux.
I say this in the most critical way possible, as someone who has built multiple Linux-based "cloud systems", and as a GNU/Linux distribution developer: I love it!
In hindsight of course it was remarkably prescient. This from a guy at a company that was built entirely around SGI at the time.
The first time I actually saw GNU/Linux powering something in production was in 2003, when I joined CERN and they were replacing their use of Solaris, and eventually alongside Fermilabs came up with Scientific Linux in 2004.
Later at Nokia, it took them until 2006 to consider Red-Hat Linux a serious alternative to their HP-UX infrastructure.
"Sir, if you have nothing to hide, why do you talk so fast, and pronounce words like a foreigner who learned English from a book?"
:)
We love you Bryan, never change.
I don't run on-prem clusters or clouds but know a couple people who do and, at large enough scale, it is a constant "fuck-shit-stack on top of itself" (to quote Reggie Watts). There is almost always something wrong and some people upset about it.
The promise of a fully integrated system (compute HW, network HW, all firmware/drivers written by experts using Rust wherever possible) that pays attention to optimizing all your OpEx metrics is a big deal.
It may take Oxide a couple more years to really break into the market in a big way, but if they can stick it out, they will do very well.
That's because they own the whole stack, from CPU to GUI and support it as a unit. That's the benefit of having a product where a single owner builds and supports it as a whole.
My impression of Oxide is that that's the level of single source of truth they are bringing to enterprise in-house cloud. So, I strongly doubt the innards would ever become customer-facing (unless the customer specifically wants that, being open source after all).
As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources. Their computers expensive enough to be very much business critical. You'll get some support by Oxide logging in remotely to customer systems and digging around, but pretty soon the customer will want to do that themselves to monitor/troubleshoot the problems as they happen.
Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
Apple is a great example of the benefits of an integrated system where the hardware and software are designed together. There are tons of benefits to that.
What makes Apple evil (IMO, many people disagree) is how everything is secret and proprietary and welded shut. But that doesn't take away from the benefits of an integrated hardware/software ecosystem.
Oxide is open source so it doesn't suffer from the evil aspect but benefits from the goodness of engineered integration. Or so I hope.
Afaik Android phones tend to have a lot more hardware than your average laptop, too (cell modem, gps, multiple cameras, gyro, accelerometer, light sensors, finger print readers)
I think the real benefit is being able to move/deprecate/expand at will. For example, want an app that would require special hardware? You can just add it. Want to drop support for old drivers? Just stop selling them and then drop (deprecate) the software support in the next release.
I fully agree about the evilness, and it baffles me how few people do!
Have you looked at the pedigree of many of the people behind the project? I don't say this because "these guys smart", but because these guys bent over backwards for their customers when they were Sun engineers. Bryan didn't write dtrace for nothing.
> Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
I think you're simultaneously over-estimating and under-estimating the people who will deploy this. There's a lot of companies who would want a "cloud in a box" that would happily plug hardware in and submit a support ticket if they ever find an issue, because their system engineers either don't have the time, desire, or competence (unfortunately common) to do anything more. The ones who are happy to start debugging stuff on their own would have absolutely wonderful tooling at their fingertips (dtrace) and wouldn't have any issue figuring out how to adapt to something other than Linux (hell, I've been running TrueNAS for the better part of a decade and being on a *BSD has never bothered me).
As we see nowadays on tablets and laptops, most OEMs are quite keen in returning back to those days, as otherwise there is hardly any money left on PC components.
Or Linux running underneath all the Java-y Android stuff.
Also, Apple is just the one who survived. Previously I'd have thought of SGI, DEC, Sun, HP, IBM, Dell some of whom survived some not.
Those three consumer products I mentioned each provide a platform for a user and business space to floroush and thrive. I expect a company doing something similar for cloud computing to want the same. But it will require some magick: momentum, money, trust. That kind of stuff, and loads of it. (With some big names behind it and a lot of FOSS they got me excited, but I don't matter.)
Their choice in foundation OS (for lack of a better term) really should not matter to any customer.
Now imagine a multi-million dollar mission critical pile of computers running on premises, and your sysadmin being able to do so.
Oxide is closer to a rack of Supermicros than AWS.
Solaris is still Solaris, as of the latest release last month. OpenSolaris hasn't been OpenSolaris in a while and is Illumos, yes.
Also there are many situations where renting, for example a flat makes a lot of sense. And there are many situations where the financials and or enabled options of owning something make a lot of sense. Right now, the kind of experience you get with AWS and co. can only be rented, not bought. Some people want to buy houses instead of renting them.
Their competition has open source firmware as well:
https://www.dell.com/en-us/blog/enabling-open-embedded-syste...
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
For a mostly open source solution, not only would you need open source BMC firmware, you must have an open source UEFI/BIOS/boot firmware like CoreBoot, LinuxBoot, Oreboot, Uboot, etc.
And Canonical played with a cluster-in-a-box all the way back in 2013-2014: https://www.zdnet.com/article/canonicals-cloud-in-a-box-unde...
You could turn it into an OpenStack cloud in ~20 mins with an automated Juju OpenStack install.
Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?
If you haven't lived through this, the answer is: nobody. Everyone points fingers at the other 4 and ignore your calls.
Back in the day you could buy a fully integrated system (from CPU to hardware to OS) from Sun or SGI or HP and you had a single company to answer all the calls, so it was much better. Today you can't really get this level of integration and support anymore.
(Actually, you probably can from IBM, which is why they're still around. But I have no experience in the IBM universe.)
This is why Oxide is so exciting to me. I hope I can be in a company that becomes a customer at some point.
Dell is a single vendor that will diagnose and fix all of your hardware issues.
With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage.
I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.
And you'll be down for weeks or months while they do it.
Off by a few orders of magnitude. Dell on-site SLA with pre-purchased spares was about 6 hours.
With Oxide, you'd be lucky to get same day service.
You're talking about replacement parts. Yes Dell is good about that.
The discussion above is asking them to diagnose and fix a problem with the interaction of various hardware components (all of which come from third parties).
There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
Neither am I.
>If you hit a "zero-day" bug that Dell has never seen it's going to take time.
If you hit a "zero-day" bug that Oxide has never seen it's going to take time.
>And somehow every large customer finds bugs that Dell certification didn't.
Yes, happens. And I'm sure the exact same will happen with Oxide, so it's not a differentiator.
It's a law of computer engineering.
In the Apollo 11 decent sequence the Rendezvous Radar experienced a hardware bug[0] not uncovered during simulation. They found it later, but until then, the solution was adding a "turn off Rendezvous Radar" checklist item.
[0] The Rendezvous Radar would stop the CPU, shuttle some data into areas it could be read, and woke the CPU back up to process it. The bug caused it to supuriously do this dance just to tell it "no new data", which then caused other systems to overload.
I've been a Dell customer at a previous company. I know for a fact that's not true.
I had a support ticket for a weird firmware bug open for two years, they could never figure it out. I left that job but for all I know the case is still open many years later.
Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell. Dell is a reseller who puts components together from a bunch of vendors and it mostly works but when it doesn't, there's nobody on staff who can fix it.
I've had support tickets open for all kinda of weird firmware, hardware, etc. bugs and they've been well resolved, even if it meant Dell just replaced the part with something comparable (NIC swap).
>Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell.
Of course they do. That's like saying Oxide doesn't know how to fix stuff because they don't design the CPU, NVMe, DIMMs, etc. Oxide is still going to vendors for these things.
At Oxide, we have been deliberate at every step, designing from first principles whenever possible. (We -- unlike essentially everyone else -- did not simply iterate from a reference design.)
To make this concrete with respect to the CPU in particular, we have done our own lowest-level platform enablement software[0] -- we have no BIOS. No one -- not the hyperscalers, not the ODMs and certainly not Dell -- has done this, and even AMD didn't think we could pull it off. Why did we do it this way? Because all along our lodestar was that problem that Dell was useless to us on -- that we wanted to understand these systems from first principles, because we have felt that that is essential to deliver the product that we ourselves wanted to by.
There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
[0] https://github.com/oxidecomputer/phbl
[1] https://github.com/oxidecomputer/illumos-gate/tree/stlouis
[2] https://github.com/oxidecomputer/illumos-gate/blob/stlouis/u...
Nonetheless, it is quite interesting what you've built, but as the end user I'm not quote convinced that it matters. Sure you can claim it reduces attack vectors and such but we'll still see Dells and IBMs in the most restricted and highest security postured sites in the world. Think DoD and such. Core/libreboot with RoT will get me through compliance the same.
The software management plane y'all built is the headlining feature IMHO, not so much what happens behind the scenes that the vast majority of the time will not have a fatal catastrophic upstream effect.
>There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
That's not what I said. There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
Set aside the childish tone ...
> Dell is a single vendor that will diagnose and fix all of your hardware issues.
There are two anecdotes here disagreeing with you, and frankly that's enough to say what you said above isn't true, not universally so. I doubt Odixe is targeting big deployment like yours, but more like theirs. Whether they will succeed is another matter, but they do have a valid sales pitch and the expertise to pull it off.
Huh, then why did Sun, Oracle, and Veritas have to set up a shared tech support center in San Jose?
"Accelerated finger pointing", said a friend who had to do business with them.
At least with Oxide there is a glimmer of hope for a better future in this regard.
Could be! Seems too early to tell though, and remains to be seen whether it pencils. Which is the whole idea of starting a new venture, no?