Outposts, too, started with a full-rack configuration only, but they eventually introduced an individual server configuration as well. It'll be interesting to see if Oxide eventually decides to serve the smaller market that doesn't have the scale for whole-rack-at-a-time.
Once installed, you plug in the network connection(s) and add power, then boot up. Then you can add your VMs and start running them.
When you cut out all that fat, you can fit a lot more muscle in its place. Then you can arrange everything on a rack scale for airflow, power, redundancy, etc.
The obvious downside is that you are locked into a single supplier for everything in the rack. Somebody please add a link here for Oxide's strategy to minimize this risk for their customers. (And feel free to correct any mistaken over-simplification.) Overall, they have seemed incredibly open to sharing in a way that allows others to play. That might just be because they would die if they didn't, but I think a lot of people are willing to accept that they are pretty all-around awesome folks trying to make the world better. And making money in the process is not a bad gig. We'll see which other vendors jump into the pool.
I wish them great success. But I'm wondering why they haven't coordinated my shipping address for their first shipment. First one's free, right?
This is not really true. Data center servers are highly optimized for density already. Like tens of thousands of hours on airflow CFD, tweaking cases and fans and internal layout, heatsink design, etc. chasing the 0.1%s. There are many significant tradeoffs to be made, but there is not a large factor improvement just sitting on the shelf due to a bunch of unused legacy leftovers.
The attack surface it leaves open is a different story.
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
More to the point with Oxide, there's layer upon layer of software cruft as well: various levels of proprietary firmware blobs driving the boot process, and then the management stack that underlies a commodity PC platform OS.
All of that is gone, replaced with a bottom-up redesign, of both the hardware and the software, with the specific purpose of running large scale modern application loads.
This goes well beyond a blade server, and is a completely different beast than a rack of DL360s.
Proof of the pudding will be in the eating I guess. Does Oxide talk about performance advantages at all, or have numbers?
If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers. At the scale of one rack, the difference is fairly small. But once you're building out cages full of racks, maybe it matters more?
And it's not just power and space (although they both matter), it's the attack surface, and the ability to actually have your hardware do what you want, and the benefits of a stack that's built for purpose, not cobbled together out of off-the-shelf parts.
It's not for everyone, to be sure, but I suspect that for its target market, it'll be a very successful product/family. As you say, we will see. I wish them well.
Not with the current state of the art.
> If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers.
I'm talking about it mattering from the starting point of blades, OCP, "cloud" systems!
The thread is about where the oxide niche is and what advantages it has over competition. The idea there is huge amount to be won on southbridge chipset and IO ports in large scale systems is simply not true. Quite amazing that people who don't understand this are posting in this thread as though they are experts in the matter.
> Not with the current state of the art.
Perhaps you could educate me: what's the state of the art that makes watts free?
>> If it didn't matter, then there'd be no interest in blades, or OCP, and all the cloud vendors would use standard rackmount whitebox servers.
> I'm talking about it mattering from the starting point of blades, OCP, "cloud" systems!
> The thread is about where the oxide niche is and what advantages it has over competition. The idea there is huge amount to be won on southbridge chipset and IO ports in large scale systems is simply not true.
I don't think anyone has asserted that there's "huge amount"s to be won from the simplified hardware? It's my view that there is a small amount to be won (more if you're coming from DL380s, and a bit less if you're replacing blades, for instance). But those small amounts do add up, and contribute to the value proposition. I think there are other parts of the Oxide product that are more compelling, but that's the point: it's the total package that you're buying, not just the missing VGA socket on the front panel.
Imagine that I'm an enterprise computing user. I currently have a few small (wrt cloud providers) datacenters, stuffed with racks of any tier 1 vendor's rack mount, blade or even OCP systems. And they're running some combination of plain VMs, perhaps some orchestration platform (k8s, or Mesos, or whatever). And it's time for my 3/5/8-yearly refresh of a bunch of those systems.
A unified, coherent, targeted product (such as Oxide) might be a compelling offer. It promises to deliver cloud-type hardware and software efficiencies at a smaller scale. It gives me the savings of not renting from AWS/Azure/etc, while not dealing with the hodge-podge of the standard PC hardware and software stack.
> Quite amazing that people who don't understand this are posting in this thread as though they are experts in the matter.
Welcome to the Internet?
Doesn't seem to be taking. The issue is that there are not a couple of watts there. There are a couple here, and that's all.
> I don't think anyone has asserted that there's "huge amount"s to be won from the simplified hardware?
They did. In this thread you are participating in even. Perhaps you didn't really read the post of mine that you first replied to, or its context?
So, this isn't really chasing the 0.1% -- it's chasing much bigger wins, and it's doing it by changing the power distribution (shelf of rectifiers to a 54V DC busbar vs. redundant AC power supplies in every 1U/2U), changing the cooling (the larger fans), changing the networking (we have a blindmated cabled backplane, obviating the need for operator cabling), etc. etc. These are not just multipliers on efficiency, but also on manageability: once physically installed, our rack is designed to be provisioning VMs orders of magnitude faster than the traditional manual rack/stack/cable/SW install.
I was replying more to the idea there's all this costly legacy IO.
I guess you aren't the first to try different geometries or power delivery or cabling either. There's been lots of little opencompute-type efforts and startups come and go. I'm skeptical there's a lot in it in a significant niche that does not already do these things, but you don't need to convince me. Although if you did want to you could show some comparative density numbers. What can you fit, 2048 cores, 32TB, and push 15kW through a rack?
And not one in particular, there's just a bunch that have sprung up around OCP over the past decade. None that I'm aware of that are doing everything that Oxide does, but we were talking more about the mechanical, electrical, and cooling side of it there -- they do seem to do okay with power density.
I thought you were going for cloud DCs rather than enterprise. Seems like a big uphill battle to get software certified to run on your platform. Are any of the major Linux distros or Windows certified to run on your hypervisor platform? Any ISVs?
> The question on density is: how can you get the most compute elements into the space/power that you've got, and cramming towards highest possible density (i.e., 1U) actually decreases density because you spend so much of that power budget cranking fans and ramming air.
The question really is how much compute power, and electrical/thermal power ~= compute power. Sure you could fit more CPUs and run them at a lower frequency or duty cycle.
> And the challenge with OCP is: those systems aren't for sale (we tried to buy one!) and even if you got one, it has no software.
OCP is a set of standards. They certainly are systems sold. I guess the nature of the beast is that buyers of one probably don't get taken very seriously, particularly not a competitor.
Yes, and using larger fans decreases the proportional there. Centralized rectifiers reduces the proportional. Google can make it all the way down to 10% overhead power. That's the point they are making.
What are the problems people are having with existing systems (like Vxrail), and how does Oxide fix those? What stories are you hearing?
For some organizations cattle-like pizza boxes or chassis with blade systems are still not cattle-like enough. By making the management unit the entire rack you can reduce overhead (at least compared to a rack of individual servers, even if they are treated like cattle).
There are vendors that will drop ship entire (multiple) racks pre-wired and pre-configured for various scenarios (especially HPC): just provide power, (water) cooling, and a network uplink.
My sibling commentor just left a great answer to your second question, so I'll leave that one there :)
From your referenced comment:
> The rack isn't built in such a way that you can just pull out a sled and shove it into another rack; the whole thing is built in a cohesive way.
> other vendors will sell you a rack, but it's made up of 1U or 2U servers, not designed as a cohesive whole, but as a collection of individual parts
What I'm curious about is how analogous or different is this cohesiveness to the days where vendors built the complete system? Is that the main selling point or there are nuances to it?
> What I'm curious about is how analogous or different is this cohesiveness to the days where vendors built the complete system?
To be honest, that was before my personal time. I was a kid in that era, using ultra hand-me-down hardware. I'd speculate that one large difference is that hardware was much, much simpler back then.
- Oxide is entirely open where Apple is extraordinarily secretive: all of our software (including the software at the lowest layers of the stack!) is open source, and we encourage Oxide employees to talk publicly about their work.
- Oxide is taking a systems approach where Sun sold silo'd components: I have written about this before[0], but Sun struggled to really build whole systems. For Oxide, we have made an entire rack-level system that includes both hardware and software: the abstraction to the operator is as turn-key elastic infrastructure rather than as a kit car.
We have tried to take the best of both companies (and for that matter, the best of all the companies we have worked) to deliver what customers want: a holistic system to deploy and operate on-premises infrastructure.