Helios: A distribution of Illumos powering the Oxide Rack
github.com
github.com
But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs"
Tell me more? :)
As I understand it, re: vertical integration, the term is actually "hyperconverged". Here, that means it's designed at the level of the rack. Like -- there aren't per compute unit redundant power supplies. There is one DC bus bar conversion for the rack. There is an integrated switch designed by Oxide. There is one company to blame when anything inside the box isn't working.
In addition, the pitch is they're using open source Rust-based firmware for many of the core components (the base board management controller/service processor, and root of trust), and the box presents a cloud like API to provision.
If the problem is: I'm running lots of VMs in the cloud. I'm used to the cloud. I like the way the cloud works, but I need an on-prem cloud, this makes that much easier than other DIY ways to achieve (OMG we need a team of people to build us a cloud...).
(That said I think other than saying "hyperconverged" your broad points are correct.)
This is fair, especially as it follows a discussion of how you're building at the rack level, which is more like a mainframe than current "hyperconverged" offerings. Recommend others read the linked post.
The products that are on the market for an AWS-like experience on-prem are still fairly horrible. A lot of times the solutions are collaborations between vendors, which makes support a huge pain (finger pointing between companies).
Or, a particular vendor might only have compute and storage, but no offering for SDN and vice-versa. This sucks because then you have two bespoke things to manage and hope they work together correct.
These companies want a full AWS experience in their datacenter, and so far this looks to be the most promising without dedicating huge amounts of resources to something like Openstack.
Took me from "why would people go with this over Dell?" to "holy shit, I'm expecting Dell to do software and make nvidia/red hat/etc/etc etc/etc etc etc help out. lol!"
* also, how destructive it is. never, ever, ever let ppl talk shit about other ppl. There's a difference between "ugh, honestly, it seems like they're focused on release 11.0 this year" and "ughh they're usless idk what they're thinking??? stupid product anyway" and for whatever reason, B made you normal, A made you a tryhard pedant
... Including the bill!
e.g., https://www.scylladb.com/2020/09/15/scylla-cloud-on-aws-outp...
Disclosure: I worked at ScyllaDB.
You buy their product once, but it only has bug and security fixes for 3 years.
Which means every business is going to need to upgrade on a cycle anyway.
Whilst I think we will see a trend back towards more on-premise hardware I don't think SaaS is going away anytime soon. And in fact it's arguably better for everyone because the software is being continually maintained.
As to why I'm personally excited: I enjoy the amount of control having such an on-prem rack would afford me, and there surely could be a great amount of cost-savings and energy-savings in many scenarios. Sometimes, you just need a rack to deploy services for your local business. I like the prospect of decentralizing infrastructure, applying all the things we've learned with IaaSes.
Of all of the 6 infrastructures I've seen, only 1 of them is half decent, with 6 dedicated teams around the datacenter working closely together (by dedicated I mean, nothing is required of them concerning the core software product that the company develops). Network, Unix/Virtu, Windows, Storage, PC, and datacenter. That's 30+ people just to run a couple big datacenters and a few more server rooms. The service was actually quite good with VMs/zones delivered under an hour and most tech issues solved in half a day. The other infrastructures were either bigger or smaller, with more or less people, and were all terrible, sometimes needing weeks of email exchanges with excel attached to get a single VM.
AWS was the dream everywhere I went for everybody. Oxide may be coming out with a product that will solve a LOT of issues. SmartOS/IllumOS has all the tech to be self-sufficient (virtualization, storage, SDN...), add support for networking and storage and you get a complete product that a handful of people can run (well, you still need a windows team in most cases but fine).
I would pay a premium just to not have to deal with HPE, DELL, etc
AWS's pricing model works kinda at their OMG eyewatering scale - aka all the custom hardware they design is highly cost optimized, but just doing custom hardware has a notable cost. This is easily covered by their scale, to make for their famous margins. [during their low scale times, they did use a good bit of HP/Dell, etc]
Oxide seems to be no different (super custom hardware) only major difference being the "in your datacenter" part. Since you own the cost of your datacenter, Oxide has to come in a lot cheaper to even compete with AWS, but how do you do that with low volume [and from the look of it not-cost optimized, but instead fairly tank-like] bespoke hardware? Feels like the pricing / customer fundamentals are going to be pretty rough here outside perhaps a few verticals.
I imagine, some of the customers will order 1-2 racks half full and over a few years possibly add a few sleds, these will probably demand great GUI/ manual experience and possibly competitive Oracle/ SAP/ MSSQL benchmarks and I can imagine Veeam integreation. Other customers such as the DoE or some big enterprise customers will order whole rows of racks and demand perfect automation options. That is just a guess.
Which should be pretty easy. I don't know the exact costs but in a previous Oxide discussion the number 1M was thrown around. If that's roughly correct, that is comfortably less than a single year of AWS bills at most startups I've been in (except the very tiny ones < 15 people).
Haven't seen any performance numbers either so admittely estimating here, but from what I know about building racks of 1U servers and knowing that Oxide is more efficient, I can believe an Oxide rack should handily outperform the AWS VMs we (the startups) were paying >>>100K/mo for.
If these numbers are anywhere in the ballpark, an Oxide rack should easily be saving quite a bit of money already by year two.
IllumOS is the OS/360 to Oxide's System/360. (It won't get that popular but it's a fair enough comparison for illustrative purposes)
I really wish I did work for an org big enough to use Oxide's gear. Not having to futz around with bogus IBM PC AT-type compatibility edifice, janky BMCs and iDRACs, hardware RAID controllers, etc, would be so unbelievably nice.
We've released a new version every two weeks post acquisition, and are continuing to develop and invest.
We also hold office hours events roughly every two weeks on Discord[2], and would love for you to stop by and ask any questions, or just listen along!
[1]: https://www.tritondatacenter.com/blog/a-new-chapter-begins-f... [2]: https://discord.gg/v4NwA3Hqay
I also think you're overestimating the extent to which make(1S) is the reason we're not more popular than Linux. There are any number of more relevant factors that make someone choose one operating system or another. Also, certainly for me personally my goal is not world domination, merely the sustainable maintenance of a body of software that helps me solve the problems that I work on, and which I enjoy using and developing as a result.
I agree we need (as do all projects!) new developers, both now, and over the long term. We work as we can to make improvements to the build process, and the documentation. We are a relatively niche project, but we do attract new developers from time to time, and we're making changes at least as rapidly as we ever have in the past. There are a number of actively maintained illumos distributions (OmniOS, SmartOS, Tribblix, OpenIndiana, and now Helios) and there are a variety of commercial interests that ship more proprietary appliances on top of an illumos base. For our part at Oxide we continue to encourage our staff to get involved with illumos development as it makes sense for them, and we try to offer resources and assistance to the broader community as well.
If you would like to contribute, we have a guide to getting started: https://illumos.org/docs/contributing/
Please, though, it's "illumos", not "IllumOS"!
Even in the early 2000s linux had a make menuconfig or make xconfig setting to build linux. And yes this is different, it's a posix distribution. Yocto was a relatively niche project as well and it also addresses the issue of building a collecting of posix applications into a big project, so does gentoo's stage.
I'm sure that at the time of it's creation OpenSolaris was ahead of its curve, but that's how many years ago? You know as well as I do that sprinkling LD_LIBRARY_PATHs here and there and then removing undocumented dot files here and there isn't really a sane way to handle such a build process for a curious third party. Most will probably drop it before it gets to that point.
There have been many many projects that have reworked their entire build architecture, some of which took years to flesh out fully.
What needs to happen for illumos to get a boost of development in the long term is:
1. first for you to acknowledge on a political level that there is an issue that needs to be addressed here, and
2. to then work with the community, and it doesn't have to be across the board, but you need to be willing to invest in some experts and some people interested in solving this, so they can grind out something that is more sane in this current world.
"Read our getting started guide" isn't really all that useful, when most of the complex issues happen after that and are often met with "this isn't how we do things".
I obviously don't have any context about the issues you were facing at the time, and I can't really figure it out based on the advice you ostensibly received. I'm definitely sorry if we have lead you astray in the past, but those are not workarounds I would encourage people to use today. If there's some aspect of the build process that requires workarounds like you're describing, it's definitely a bug and we'll fix it when we're made aware as best we can.
As for the rest of it, I think you're putting the cart before the horse on some level. An operating system is a large and complex thing to work on, regardless of whether it's built with make or ninja or bazel or whatever other build tool.
The Rust toolchain is another similarly complex body of software, which also has a large and at times inscrutable build process. I know because I have personally contributed to it, and had to figure out how to get it to work. Rust obviously has more active contributors than illumos, but it also has vastly more active _users_ -- it is a body of software that has broad applicability to many people and the work they do.
For illumos to continue to succeed as an actively maintained project, what we need to do is continue to inspire _users_ to want to use it. Nobody wants to work on an operating system they don't personally need to use at all. We draw contributions today from a mixture of community driven distributions making fixes or adding features, and by people employed by companies like Oxide who have a vested economic interest in the deployment of the software.
None of this is to say that we're perfect, or that we're not trying to improve things. Just that we're trying to put build system improvements in the proper context amongst all the other work there is to do with our limited resources. It's probably more important that we have support for new Intel client NICs like you would find in a modern desktop system, for example, than it is that we replace make. It's important that we continue to add system calls and libc facilities that other platforms have adopted in order to ease software porting. It's important that we continue to maintain modern JDKs and Python and Go and Rust and C/C++ compilers. It's important that we keep up with security issues and the endless stream of mitigations imposed by the sieve-like nature of speculative CPUs.
There's actually quite a lot of stuff going on for us all the time, and we do still find time to improve the build system. If you have more specifics in mind, that's fantastic and we'd love to here about them concretely! I would encourage you to channel your enthusiasm into writing an illumos project discussion (IPD) describing the issues you see and the work you'd propose to sort them out! You can see some examples of existing IPDs at https://github.com/illumos/ipd
And as ever, if you hit issues in the build as it stands, please file bugs! We can't fix things we haven't heard about.
Cheers.
I remember buying the OpenSolaris Bible in 2008, getting really excited to dig into my second Unix (after FreeBSD). And then, the Sun went down on me... and I stuck with Ubuntu 10 years.
For a while Nexenta had an Ubuntu running on the OpenSolaris kernel.
I recently wrote a guide [1] how to use proxmox with ZFS over iSCSI so you can use the snapshot features from a SAN
[1] https://blog.haschek.at/2023/zfs-over-iscsi-in-proxmox.html
At its peak I reached out multiple times to Joyent to fix their EFI support for virtualization. The Danube team had similar experiences with them, working on live migrations for VMs, and a few months back I did a rebase of the platform image to a more recent illumos stack.
Two of the fundamental issues with Illumos is that they don't seem to understand that they need to fix the horrendous platform build to get community support to keep up with the pace of development of other OS's. The platform build is a huge nasty mess of custom shell scripts, file based status snapshots, which includes the entire userspace in the kernel build. Basically if your openssl version is out of wack the entire thing will fail. Not because it has to, but because it was never adapted to modern needs of someone just wanting to hack on a kernel. It's fixable, but I don't see any desire to fix it, and even if that desire eventually shows up it might just be too little, too late.
Same for me. Oxide is the only company I know that I'd really love to work for. Similar (I think, observing from the outside) to Sun. That's what I dream about.
Unfortunately their pay structure is such that I can't afford it, with a family to support. Maybe when the kid is out of university, if I don't need much income anymore, I can fulfill the dream.
Thought I was the only one :P
I really miss Sun.
There's nothing in the world I miss so much as Sun.
I've done many startups post-Sun and there's been good highs and many lows but, there's nothing like a true hard-core tech company like Sun.
There basically are no tech companies anymore, other than Apple, but they are mass-market consumer oriented which is not interesting.
Kind of. Apple focuses their high end gear on creative professionals. Us Unix geeks have much more modest needs, which are often satisfied by the average uninspired Dell design. Still, Apple has a decent Unix underneath all that glitter. At the same time, there are almost no desktop-friendly Unixes besides the free crowd. HP and IBM have given up on the Unix workstation market eons ago. IBM’s POWER gear can crush the best Xeons and Epycs, but they have nothing to compete with the “good enough” low end.
It’s a shame Oracle doesn’t offer Solaris on their cloud the same way IBM offers AIX (and Z, which, surprisingly, is a certified UNIX as well) on theirs.
Had. That's what pulled me in to Apple laptops after spending the 90s convinced I'd never use a Mac.
With OS X, suddenly it was BSD, but with a Mac GUI! Cool. When OS X (10.0) came out I quickly bought a mac first time ever.
But Apple has spent the last 20+ years making OSX less and less BSD, locking out more and more core functionality into obscure nonstandard behavior.
Still own several Sun CDs they used to give away to developers.
What they can’t do as well as a deskside Ultra Enterprise is to look impressive.
The differentiator from virtually all existing on-prem cloud products is that we are a single vendor who has designed the hardware and software (which is as open source as we can possibly make it, by the way, hence announcements like this) to work well together. Most products combine various other products from various vendors, and are effectively selling you integration. We believe that that leads to all kinds of problems that our product solves.
Another factor here is that we only have two SKUs: a half rack and a full rack. You don't buy Oxide 1U at a time, you buy it a rack at a time. By designing the entire rack as a cohesive unit, we can do a lot of things that you simply cannot do in the 1U form factor. There is a running joke that we talk about our fans all the time, and it's true. Because our sleds have a larger form factor than a traditional 1U, we can use larger fans. This means we can run them at a lower RPM, which means power savings. That's the deliberate design choice. But we also have gained accidental benefits from doing things like this: lower RPM also means that our servers are way quieter than others. That's pretty neat. Some early prospective customers literally asked if the thing is on when it was demo'd to them, because it's so quiet. Is that a reason to buy a server? Not necessarily, but it's just a fun example of some of the things that end up happening when you re-think a product as a whole, rather than as an integration exercise.
I know of the technical benefits of illumos over linux, but does that actually matter to the customers who are buying these? Aren’t they opening a whole can of worms for ideology/tradition that won’t sell any more computers?
As someone who runs Linux container workloads, the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for.
It's not like we specifically say "oh btw there's illumos inside and that's why you should buy the rack." It's not a customer-facing detail of the product. I'm sure most will never even know that this is the case.
What customers do care about is that the rack is efficient, reliable, suits their needs, etc. Choosing illumos instead of Linux here is a choice made to help effectively deliver on that value. This does not mean that you couldn't build a similar product on top of Linux inherently, by the way, just that we decided illumos was more fit for purpose.
This decision was made with the team, in the form of an RFD[1]. It's #26, though it is not currently public. The two choices that were seriously considered were KVM on Linux, and bhyve on illumos. It is pretty long. In the end, a path must be chosen, and we chose our path. I do not work on this part of the product, but I haven't seen any reason to believe it has been a hindrance, and probably is actually the right call.
> the fact that this is fundamentally not-Linux (yes I know it runs Linux binaries unmodified) would be a reason against buying it, not for.
I am curious why, if you feel like elaborating. EDIT: oh just saw your comment down here: https://news.ycombinator.com/item?id=39180814
This would allow workloads on Oxide to run their own VMs, to safely use PCIe devices without dealing with interrupt redirection, etc.
But I do wonder why cloud and cloud-like systems aren’t more aggressive about splitting the infrastructure and tenant portions of each server into different pieces of hardware, e.g. DPU. A DPU could look DPU could look like a PCIe target exposing NVMe and a NIC, for example.
Obviously this would be an even more custom design than Oxide currently has, but Oxide doesn’t seem particularly shy about such things.
It is true that the number of developers that know Linux is larger than the ones that know illumos. But this is also true of the number of developers who know C as the ones who know Rust. Just like some folks need to be onboarded to Rust, some will need to be onboarded to illumos. That is of course part of the tradeoff.
You're much better off hiring people who are capable of learning, and then giving them the opportunities to learn and advance their knowledge and skills.
You're better off hiring experienced people who are highly productive. If they're highly productive with one stack, it makes no sense to change their stack so they're no longer productive, or hiring people who aren't familiar with it and waiting for them to become productive.
There's nothing wrong with using old, well established things. They're quite often better than new things. As long as they're still supported, just use whatever builds a working product. It's the end product that matters.
The difference between Japanese and English is much, much bigger than the difference between one Unix OS and one Unix-like OS. This is a remarkably disingenuous argument. If you really don't understand the difference in scope, there's no point in discussing anything with you because you've managed to disprove your opening sentence with yourself as the counterexample.
(Japanese isn't so radically different from English, it mostly just has more words for more contexts. In many ways it's simpler than English. It would be harder to go from Java to Haskell, with their many different language paradigms)
That's not the big concern though. The big concern is whether vendor integration and certification becomes a stumbling block. You can hire any monkey to write good-enough code, but that doesn't give you millions in return. Partnerships with vendors and compliance certifications can give you hundreds of millions. The harder that is, the farther the money is. A totally custom, foreign stack can make it harder, or not; it depends how they allocate their human capital and business strategy, whether they can convince vendors to partner, and clients to buy in. Anything very different is a risk that's hard to ignore.
That being the case, knowing yet another UNIX cousin isn't that big deal.
It also was not discussed as a factor in the RFD.
As for why an illumos derivative and not something else, we expanded on this a bit in our Q&A when we shipped our first rack[1] -- and we will expand on it again in the (recorded) discussion that we will have later today.[2]
[0] https://hubris.oxide.computer/
The only thing I miss about Twitter Spaces is that you could listen the morning after.
I am looking forward to the crucible episode. It sounds like it could be a startup on its own, it wouldn't be the first distributed file/ storage system company.
Or one goes the route of 99% of the IoT/etc vendors, and never update the base OS and pray that there aren't any active exploits targeting it.
This is why a lot of medium-sized companies cried about Centos, which allowed them to largely stick to a fairly stable platform that was getting security updates without having to actually pay/run a full blown RHEL/etc install. Every ten years or so they had to revisit all the dependencies, but that is a far easier problem than dealing with a year or two update cycle, which is too short when the qualification timeframe for some of these systems is 6+ months long.
So, this is almost exclusively a Linux problem; any of the *BSD/etc. alternatives give you almost all of what Linux provides without this constant breakage.
This weighed heavily in our own calculus, so I'm glad you brought it up!
giving limited resources of the dev team it may lead to limited support of the system outside of the narrow set of officially supported/certified hardware with that support falling behind on modern hardware, as it happened with Sun, and vendor lock-in as a result into overpriced and low performing hardware.
There is a reason that back then at Solaris dev there was a joke about embedding Linux kernel as a universal driver for Solaris kernel in order to get reasonable support for the hardware around.
The Oxide hw is using available AMD SKUs for CPU.
FreeRTOS, Nuttx, Zephyr, mbed, Azure RTOS,...
Embedded isn't limited to devices equal or less powerful / expensive than the Raspberry Pi.
This is no different from Azure Host OS, Bottlerocket, Flatcar or whatever.
This maters to them, as knowing the whole stack, some of the kernel code is still theirs from Sun days, and making it available matters to the customers that want source code access for security assement reasons.
This is the substrate upon which you will run any virtualized infrastructure.
Disclaimer: Former Googler, Cloud Support
Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?
Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility.
Rest of the bigger ones is mainly Linux with occasional Xen and such, but sometimes you can encounter non-trivial VMware deployments.
Yup. eg with Xen the hypervisor wasn't Linux, even if the privileged management VM (dom0) was Linux (or optionally NetBSD in the early days). The very small Xen hypervisor running on the bare metal was not a general purpose OS, and didn't expose any interface itself - it was well hidden and relied on dom0 for administration.
https://techcommunity.microsoft.com/t5/windows-os-platform-b...
I think the only provider where that would make sense would be Microsoft, where they have their own OS.
In contrast, it appears that Oxide is committing to open-source the equivalent pieces of their virtualization platform.
https://docs.aws.amazon.com/whitepapers/latest/security-desi...
However, if we are considering code quality, which I consider important if you are actually going to be maintaining it yourself as oxide will have to do since they need customizations, then most of the proprietary Unix sources are just superior imo. That is, they have better organization, more consistency in standards, etc. The BSDs are slightly better in this regard as well, it really isn't a proprietary vs open source issue, it's more about the insane size of the Linux kernel project making strict standards enforcement difficult if not impossible the further you get from the very core system components.
Irregardless of them being ex-Sun (and I am not ex Sun), if I needed a custom OS for a product I was working on, Linux would be close to the last Unix based OS source tree I would try to do it with, only after all other options failed for whatever reason. And that's not even taking into account the licensing, which is a whole other can of worms.
1. No support for nested virtualization, so running a vm inside your vm is not available. This prevents use of projects such as kubevirt or firecracker on a Linux guest, and WSL2 on a Windows guest.
2. No GPU support
If the base hypervisor was Linux, it would be way more capable for users it seems. I also wonder if internally Linux is used for development of the platform itself so they can create "virtual" racks to dogfood the product without full blown physical racks.
With all that said, I do not know the roadmap and admittedly there are already quite a few existing platforms built on kvm, so as their hypervisor improves and becomes more capable it could potentially become strategic advantage.
Developers at Oxide work on whatever platform they'd like, as long as they can do their work. I will say I am in the minority as a Windows user though, most are on some form of Unix.
> so they can create "virtual" racks to dogfood the product without full blown physical racks.
So one of the reasons why Rust is such an advantage for us is its strong cross-platform support: you can run a simulated version of the control plane on Mac, Linux, and Illumos, without a physical rack. The non-simulated version must run on Helios. [1]
That said we do have a rack in the office (literally named dogfood) that employees can use for various things if they wish.
1: https://github.com/oxidecomputer/omicron?tab=readme-ov-file#...
> I will say I am in the minority as a Windows user though, most are on some form of Unix.
Now i'm imagining Helios inside WSI - Windows Subsystem for illumos
> Now i'm imagining Helios inside WSI - Windows Subsystem for illumos
That would be pretty funny, ha! IIRC something about simulated omicron doesn't work inside WSL, but since I don't work on it actively, I haven't bothered to try and patch that up. I think I tried one time, I don't remember specifically what the issue was, as I don't generally use WSL for development, so it's a bit foreign to me as well.
Man you can't let Brain live that one down can you?
:)
I mean... WSL2 is just hyperv with some integration glue, and illumos isn't Linux but unix is unix; that might well be doable.
We certainly understand that there's space in the market for a GPU-focused product, but that's a different one than the one we're starting the company off with. There's additional challenge with how we as a company desire openness, and GPUs are incredibly proprietary. We'll see what the future brings. Luckily for us many people still desire good old classic CPU compute.
At $work I'm running SmartOS servers with GPU passing to a ubuntu bhyve for the occasional CUDA compute and it works wonderfully. Wonder if similar could be possible with Helios?
Would you say that Oxide would have chosen Illumos if he wasn't part of the company?
I don't know how to respond to this question, because to me it reads like "if things were completely different, what would they be like?" I have no idea if you could even argue that a company could be the same company with different founders.
What I can say is that this line of questioning still makes me feel like you're implying that this choice was made simply based on preference. It was not. I am employee #17 at Oxide, and the decision still wasn't made by the time I joined. But again, the choice was made based on a number of technical factors. The RFD wasn't even authored by Bryan, but instead by four other folks at Oxide. We all (well, everyone who wanted to, I say "we" because I in fact did) wrote out the pros and cons of both, and we weighed it like we would weigh any technical decision: that is, not as a battle of sports teams, but as a "hey we need to drive some screws: should we use a screwdriver, a hammer, or something else?" sort of nuts-and-bolts engineering decision.
I'm not saying otherwise.
In fact, when I wrote my original comment, I actually rewrote it multiple time to be sure it wouldn't suggest I was thinking it was some sort of irrational decision (that's why I added the “it's a good fit for what they are doing”), but given your reaction it looks like I failed. Written language is hard, especially in a foreign language, sorry about that.
Heck, there's a great little mistake of communication in the title: this isn't just "intended" to power the rack, it does power the rack! But they said that because we said that in the README, because that line in the README was written before it ended up happening. Oops!
Bryan is just one out of several illumos experts here. If none of those were around, sure, maybe we wouldn't have picked illumos -- but then we'd be unrecognizably different.
I came into Oxide with a Linux background and zero knowledge of illumos. Learning about DTrace especially has been great.
https://www.brendangregg.com/ebpf.html
https://www.brendangregg.com/blog/2018-10-08/dtrace-for-linu...
Not the case on FreeBSD/Windows/macOS, where it's all or nothing. Still, on FreeBSD you just need root, on macOS you need to boot into a special kernel mode which doesn't even work on modern Apple Silicon machines.
Is it that it runs Linux binaries unmodified or that it runs vms and manages VMs which run Linux, and as an end-user, that's what you run your software in?
VM per container seems like a waist unless you need that extra isolation.
I agree that VM per container can be wasteful - though something like Firecracker at least helps with start time.
They might get into adding more higher level software eventually depending on what costumers want.
I have no reason to believe Oxide would be anything but responsive and effective in supporting their systems, but introducing bespoke software this deep in the stack severely curtails my options if things get bad.
> illumos is a Unix operating system
Is illumos an actual Unix (like macOS) or a Unix-like OS (like GNU/Linux)?
> It is based on OpenSolaris, which was based on System V Release 4 (SVR4) and the Berkeley Software Distribution (BSD). Illumos comprises a kernel, device drivers, system libraries, and utility software for system administration. This core is now the base for many different open-sourced Illumos distributions, in a similar way in which the Linux kernel is used in different Linux distributions.
https://www.opengroup.org/openbrand/register/
so it can't use the UNIX™ trade mark.
But it's got the AT&T Unix kernel & userland sources contained in it.
PDP-11 Unix System III: https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/ut...
IllumOS: https://github.com/illumos/illumos-gate/blob/b8169dedfa435c0...
- Derived from Bell Labs unix source
- Legally allowed to use the UNIX trademark (AKA certified Unix)
- A unix-shaped OS (similar but not 100% the same as POSIX complacence)
and those things are basically independent. Most GNU/Linux are unix-likes but not derived from original unix code or certified, but there's been 1-2 that did get certified. The BSDs are (now quite distantly ) derived from unix source but not certified (although ex. UnixWare is IIRC). Solaris was all 3 but OpenSolaris and now illumos are obviously unix-like and still based on the original code but not certified UNIX™.
(Take all this with a grain of salt; I'm typing this all from memory and IANAL)
> Oxide customers include the Idaho National Laboratory as well as a global financial services organization. Additional installments at Fortune 1000 enterprises will be completed in the coming months.
Lazer focus on the first set of customers that will help you cross the chasm. Only then mass market.
Some examples:
x86 owned the desktop, workstation and laptop world for a long time. So everyone targeted x86, which made x86 the default in the datacenter. It was hard for ARM to break in and it mostly happened when AWS did it by fiat. If ARM had made some loss-leader actually useful laptops and workstations available, it might have happened sooner.
But x86 largely didn’t deploy AVX-512 in client machines, so people who wrote libraries only used it for fun or benchmarking, so it wasn’t widely used, and most users flubbed it anyway. (And might have gotten it right if they had the hardware on their desk.)
People target Nvidia datacenter GPUs. But people have targeted them for a long time, because they have them in their gaming machines too.
Xilinx used to push free academic gear quite hard, because that was a big lead into people learning how to use their gear.
So, if I were giving Oxide straightforward sales advice, absolutely don’t get distracted with small systems. But maybe, if Oxide thought of it as lead generation, Oxide should do it anyway. If I could buy something small enough to be affordable but big enough to be useful [0], I might get one. And I’d target it with my own stuff, and fix bugs, and evangelize it at little cost to Oxide.
[0] For me, maybe 100-150TB of spinning rust (or cheap NVMe or the ability to attach a JBOD), plus anywhere from 4-64 cores, in a format that works on 120V and fits in, say, 16U or less, at a credible price point, would be quite likely to net Oxide a sale. (Just one sale but still!) It could be sold as a developer thing, and there would be absolutely no expectation that it would perform like the real thing. If I found it awesome, I might buy a couple more. But I would also use it and make things work on it and talk about it, and if a whole bunch of people did this, Oxide might get a bunch of real sales.
(Also, I get the idea behind two SKUs, but can buyers at least configure storage and compute separately? Different workloads need radically different ratios.)
> (Also, I get the idea behind two SKUs, but can buyers at least configure storage and compute separately? Different workloads need radically different ratios.)
Right now, this early: no. Sleds have compute and storage located together, so the unit of customization is currently "number of sleds in the rack" which according to https://oxide.computer/product/specifications apparently is currently three, not two, at the moment: 16, 24 or 32 sleds.
You are right that these need to be different for certain customers and workloads, we just aren't ready to support those just yet. We'll get there. Same issue, different aspect.
But Oxide is small and shouldn’t listen to me unless a customer asks for this.
A future of developers and CTOs who grew up with Oxide!
I get that in theory… and it makes sense most of the time. I’m not sure it does this time though. You don’t exactly “target” Oxide as an OS or platform. Rather, you use it to run VMs on. Those VMs are whatever you want. Other than that, I’m not sure what else having a home-lab version of Oxide would look like.
A different competitor to Proxmox?
Yes, you can do slimmed down cloud deployments, but you're still not running the (actual) S3 or EC2 backends at home.
At most, you’re talking a K8s based deployment (for a workload that could scale up or down). But that’s also not at the level of working with Oxide directly. And I doubt Oxide wants to get into the business of selling access to their own public cloud.
Looks like a cloud like experience on your own hardware.
If only it were as cheap as dell...
That goes double if it's your CTO...which is exactly what ended up happening with us.
I'm not saying "no, never", but clearly "no, not right now".
I would not get surprised if Oxide next customers were a few giant banks and funds.
They are able to identify, and most importantly, quantify risk in a way that many businesses cannot.
Consequently, they're able to take risks with new hardware/software that other companies shy away from.
Some (mostly tooling) questions:
- Why gmake? Especially since dmake is needed later anyways?
- Instructions say run rustup with bash explicitly; is that a defect in upstream, or is the local sh not completely posix compatible?
- How is this developed internally? Do Oxide folks run illlmos workstations or is this all developed in Virtual machines or SSHed to servers?
- Why MPL? GPL compatibility?
> Do Oxide folks run illlmos workstations or is this all developed in Virtual machines or SSHed to servers?
I wrote about this topic here: https://news.ycombinator.com/item?id=39181727
That said, some folks certainly run illumos on a workstation.
> Why MPL? GPL compatibility?
On MPL: https://news.ycombinator.com/item?id=39181844
That said in that comment I didn't really speak to the "why." We feel like it's a good compromise in the possibility space: more copyleft than BSD, but also less restrictive than the GPL.
And what would happen if, for whatever reason, a company can no longer purchase Oxide racks, will it need to start over its infra, or can it just build around Oxide hardware?
If you're interested in a distribution targeting end user use of illumos on servers, I would absolutely recommend looking at OmniOS! Helios is very closely based on OmniOS r151046 LTS, and we use that LTS of OmniOS release directly for non-Oxide-rack infrastructure systems inside Oxide as well.
I'm rooting for more mature OS diversity, I just have no idea who the end users would be and what their needs would look like.
As for why they used Illumos, many of the people came from Sun, Joyent, etc. so there's an obvious bias. However they do have a compelling reason that this is not an IBM compatible x86 personal computer, there's no BIOS, no UEFI, no traditional BMC, as far as I can tell they've removed as much proprietary firmware and binary blobs as they could possibly remove, while still using modern x86.
Each sled has a service processor and a hardware root of trust that directly boots the CPU, loads the AMD training blob, and boots the OS. It would be difficult to upstream the changes required to do that into a Linux or BSD for a computer only you currently have. So you'd have to maintain your own downstream fork, there is no one else responsible for the robustness of the OS, so it might as well be OS that you have had to support and develop for years.
I'd imagine as broadcom slowly destroys vmware's market share vendors will look to alternatives, but I doubt b-hyve is even a blip on their radar at this point.
bhyve, unlike other "famous" hypervisors is pretty stable, has good enough virtualized drivers (altho I'm sure Oxide has made it better) and can boot a VM with 1.5 TB of RAM and 240 vCPU[1]. Something I was not able to do with anything other than bhyve.
I know this is HackerNews, so I have to say it, marketing != engineering. Just because the FreeBSD project's marketing suck, doesn't mean engineering is bad. usually it better than the mainstream ones.
1: https://antranigv.am/posts/2023/10/bhyve-cpu-allocation-256/
A lot of the vendor-provided OVAs cut out a bunch of hardware support with the assumption that they only need to support vmware emulated hardware.
There's a good argument that your servers in the cloud don't need to be on the same OS, as long as you can hire enough talent to work on them.
I somehow landed on their podcast because it covered <whatever the hell I thought was interesting at that moment>.
The podcast is for me amazeballs marketing - it does everything but sell their product (might be a good idea to add a pitch in for each out-tro!)
I mean they talk about it, like “we had such a tough time getting the compiler to do something something and then veer off to discuss back in the day stories.
Ah never mind. Keep talking guys hope it works out
'Oxide and Friends' however isn't really what I would consider a podcast, but a recording of live "spaces" or group calls, beginning on Twitter and now happening in Discord. IMO it's not really best consumed as a podcast, but rather to participate in live. If you tune in live you'll pick up on the vibe of the recordings a lot better.
MPL 2.0 is an interesting license choice, for an operating system.
EDIT: why the downvotes?
> For any new Oxide-created software, the MPL 2.0 should generally be the license of choice. The exception to this should be any software that is a part of a larger ecosystem that has a prevailing license, in which case that prevailing license may be used.
EDIT: I also am confused about why you are downvoted. Are there any major operating systems distributions that are MPL licensed? I can't think of any off the top of my head. Beyond that it's a simple question.
By the way, you can browse public RFDs here: https://rfd.shared.oxide.computer/
I didn't include any links to any RFDs in my comments today because I have only been referencing non-public ones.
How does that work if the prevailing is BSD/MIT/ISC?
You're saying that Oxide can then be licensed under BSD/MIT/ISC?
> For example, Rust crates are generally dual-licensed as MIT/Apache 2.
We often produce components that we share with the broader open source world. For example, dropshot[1] is our in-house web framework, but we publish it as a standalone package. It is licensed under Apache-2.0 instead of MPL 2.0 because the norm in the Rust ecosystem is Apache and not MPL.
> You're saying that Oxide can then be licensed under BSD/MIT/ISC?
I am saying that we do not have one single license across the company. Some components are probably BSD/MIT/ISC licensed somewhere, and I guarantee that some third party dependencies we use are licensed under those licenses. That's different from "you could choose to take it under BSD," which I didn't mean to imply, sorry about that!
“And because any conversation about open source has to address licensing at some point or another, let’s get that out of the way: we opted for the Mozilla Public License 2.0. While relatively new, there is a lot to like about this license: its file-based copyleft allows it to be proprietary-friendly while also forcing certain kinds of derived work to be contributed back; its explicit patent license discourages litigation, offering some measure of troll protection; its explicit warranting of original work obviates the need for a contributor license agreement (we’re not so into CLAs); and (best of all, in my opinion), it has been explicitly designed to co-exist with other open source licenses in larger derived works. Mozilla did terrific work on MPL 2.0, and we hope to see it adopted by other companies that share our thinking around open source!”
https://bcantrill.dtrace.org/2014/11/03/smartdatacenter-and-...
Also discussed around 38 minute of https://youtu.be/Zpnncakrelk?si=DkSW6CM_MS-q1Gyd
Although not explicitly stated there are like deeper roots here “The one important exception to these generalizations is Sun Microsystems' CDDL, which was a true improvement on MPL 1.1, and which continues to cover a substantial amount of important open source software. … I encourage Oracle, the current CDDL steward, to consider relicensing its CDDL code under MPL 2.0, which is as worthy a successor to CDDL 1.0 as it is to MPL 1.1.” from Richard Fontana’s article at the time of the MPL 2.0 release, https://opensource.com/law/12/1/the-new-mpl
With its compatibility with strong, older copyright licenses I’m surprised the license has not had more widespread adoption. It is a not too hot, not too cold porridge of a file level copyleft and CYA OSS license with the strong backing of Mozilla.
> explicit warranting of original work obviates the need for CLAs
What do those terms mean?
Nothing has changed with regards to our anti-blob and pro-open source stances. I am not sure what you're referring to here.
> with unnecessary lock-in.
What lock-in are you referring to here? The way that things run on the rack is via virtual machines, you can run virtual machines on many providers. We even have a terraform provider so that you can use familiar tools instead of the API directly, if you believe that is lock-in (and that stuff is all also fully open source).
To be clearer regarding my questions:
- What happened to Project X (supposedly coreboot++ for latest AMD CPUs)? It seems dead, despite being more reported on than Oxide's attempts in working with AMD (to achieve the same outcomes, presumably - what's the difference?). Loads of well meaning people have approached this with virtue, innocence and skills; perhaps another approach is needed that fully respects the dynamic between the user, the chip manufacturers and the governments and banks they're in debt to.
- Does Oxide attempt to sandbox, completely remove or 'verify as benign' aspects like the PSP? For example, if someone could verify that the PSP cannot possibly be affected over the network, then peace of mind could be more affordable regarding things like supply chain attacks and bad actors with AMD/Intel/Apple management engine secrets.
Not referring to software lock-in, just hardware. And it isn't very nefarious like other hardware lock-in (serialization, see Rossmann Group). Just hardware on the rack-level: replacing oxide gear & upgrading oxide gear (not sure about repair, that could be easy). And if the offering were of a less blobby architecture, then many of us would be happy to pay a bit more for the hardware as a system. However, if the hardware platform is FOSS, then it won't be unnecessarily difficult to mix and match and integrate the Oxide gear with other DC-class gear.
> What happened to Project X (supposedly coreboot++ for latest AMD CPUs)?
I don't recall what you're referring to specifically, maybe this was a thing before I started at Oxide. I do know that we deliberately decided to not go with coreboot. I believe the equivalent component would be phbl[1]. It boots illumos directly. Bryan gave a talk about how we boot[1][2] with more reasoning and context.
> Does Oxide attempt to sandbox, completely remove or 'verify as benign' aspects like the PSP?
The general attitude is still "remove or work around every binary blob possible," but the PSP is unfortunately not able to be worked around.
> However, if the hardware platform is FOSS
We fully intend to do this, by the way. Just haven't yet. It'll come.
1: https://github.com/oxidecomputer/phbl
2: https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
This seems to make no mention of Oxide at all. Perhaps you're connecting two different unrelated organizations together as Oxide appears to have never had any relation to it. I think you're just perhaps confused about what the situation is.