The Cloud Computer
oxide.computer
oxide.computer
They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration problem. They integrated it. It comes with all the features they expect you to want if you wanted to build your own cloud.
Also, they wrote the software. And it's all open source. So no "sorry but the third party vendor dropped support for the bios". You get the source code. Even if Oxide goes bust, you can still salvage things in a pinch.
Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.
They sell rack-as-compute.[0] Their minimum order is one rack: You plug in power and network, connect to the built-in management software (API), and start spinning up VMs.
[0] With built-in networking and storage.
This achievement is clearly worth a lot to people.
It's nice, it's just nothing new.
I believe the episode name was "Virtualizing Time"
"no cables no assembly just cloud" wtf is that
Size of doorways, weight bearing capacity of floors, electrical service parameters, environmental conditions, etc.
e.g. Does it actually handle electrical voltage fluctuations of +/- 1V, or whatever is advertised?
The guaranteed parameters of a fully set up machine:
Minimum performance metrics, software compatibility with whatever the sales department promised, maximum power draw, etc.
e.g. Can it reliably hit X metric (FLOPS, IOPS, Integer calculations, etc.)?
And so on.
Or does the customer need to take on some risk and hazard associated with installation, configuration, initial boot up, etc.?
e.g. If someone buys with the intention of using it up to X FLOPS, and the machine only delivers Y FLOPS once it's all said and done, what happens?
Otherwise I would imagine it would be a major selling point and be advertised publicly.
But if it is a complicated enough internal issue for that not to be the case, then my apologies for pestering.
The comment you replied to was not questioning the value of integrated cabling. It was pointing out that the product description on the site does not make sense.
"Cloud computer" sounds like a server you rent from AWS. It's kind of like calling Rust "cloud compiler."
If you choose to use words that your audience doesn't understand, or even worse understands to mean the opposite of what you want them to mean, it's a good idea to explain these words immediately using conventional words with conventional meaning. The comments by throw0101a did that.
The product seems really cool, but there is no way I would've understood what it was from the website.
And for what it's worth, I don't think you need to explain what's happening to Steve, it seems to me that he understands perfectly well. To me you come across as being rather condescending and in my opinion Steve is being commendably polite in response.
Cloud as a term is pretty undefined/overloaded. It makes sense to me if I think about 'cloud' computing as API-based provisioning on pooled resources.
No cables, except for a few cables.
To be super clear about it, this is referring to not needing to cable up all of the individual sleds to the rack upon installation. It doesn't mean that we recommend connecting a rack of compute to your data center via wifi.
I think their description conveys what it does just fine for the target audience.
When people hear cloud, it means that aspects such as electricity costs, electricity stability, Internet, bandwidth, fire protection, safety, etc etc are abstracted away.
Oxide IS on-premise, right? The website is very vague and wishy-washy.
Ehrm. What is that exactly?
Are you alluding to cute design, different user interface? Or ditching then common PC component modularity? "Thinking differently"?
One difference is that Oxide development was done in the open and they don't seem hell bent on creating a closed ecosystem. (Yet, at least)
You are also correct that we diverge from Apple in other ways, such as our commitment to openness, rather than secrecy.
The first iMac famously made it easy to connect to the Internet; The 'i' in iMac was for "Internet". Its setup manual was a couple of pages long, mostly pictures and IIRC, just 37 words.
Sun did that experiment:
* https://en.wikipedia.org/wiki/Modular_data_center
Also:
* https://www.deltapowersolutions.com/en/mcis/data-center-solu...
* https://atos.net/en/solutions/high-performance-computing-hpc...
* https://www.zelladc.com/zella-max
Search for "shipping container data centre" or "containerized datacenter".
I’m actually extremely impressed. I want one. I haven’t worked in a data center in years, but I’d be tempted to do it again just to get my hands on one.
Oxide feels like it'll be that "integrated" experience, but with the added benefit of software freedom.
Also https://artemis.sh/2022/03/14/propolis-oxide-at-home-pt1.htm...
Does illumos run on ARM?
I wish IBM would shrink their LinuxONE Rockhopper 4 Rack Mount down to at least an "under the desktop" model. To my knowledge, IBM still makes quality products and has excellent customer service. They have fun names too (Rockhopper and Emperor are types of penguins!) and they even have 3D models of their rack mount cloud computers with shadows. [3] In fact, when I first read about Oxide a year or two ago, I searched for "IBM cloud server", and left it at that. So IBM, could you please send someone from the LinuxONE down to Boca Raton to create our new PC? :) Thanks!
P.S. I would even accept a shrunken 370. :)
[1] https://en.wikipedia.org/wiki/IBM_Personal_Computer
[2] https://archive.org/details/byte-magazine-1981-01/page/n313/...
[3] https://www.ibm.com/demos/it-infrastructure/ux/product.html#...
Proxmox and a full rack of Supermicro gear would not be as sophisticated, but end result is pretty much the same, with I imagine far far better bang for buck.
I like it, but it doesn't seem like a big deal or revolutionary in any way.
When it breaks, good luck!
You're no worse-off with Oxide from that perspective. Their open source firmware means that thr opportunity to pay somebody else to support you at least exists.
Even small shops can use bad experience as leverage for credits and discounts, especially if the vendor has account managers. This is one of the (few) benefits of having a human involved in invoicing vs. self-serve.
> when any of that breaks, good luck!
The premise is that you don't need luck, you can call Oxide. As you said, they wrote all of it, so they own all the interaction so they can diagnose all of it.
When I call Dell with a problem between my OS filesystem and the bus and the hardware RAID, there's at least three vendors involved there so Dell doesn't actually employ anyone that knows all of it so they can't fix it.
Sure, Oxide now needs to deliver on that support promise but at least they are uniquely positioned to be able to do it.
The rest comes down to execution. Sure, we all have high hopes for Oxide. Sure, we all hate established players like Dell.
Just about any company will sell your company a support contract.
The more interesting question is, can they back it up with action when push comes to shove? I suspect most people have plenty of stories of opening support tickets with big name vendors that never get resolved. And through the grapevine you find out that they won't fix it because they can't fix it. They might not even have access to the source code or anyone on staff who has a clue about it because it came from who knows where. Sales is happy to sell you the support contract but it doesn't mean your problems can be fixed. BTDT.
From listening to the Oxide podcasts, my impression is that Oxide actually can technically fix anything in the stack they sell, which would make them vastly different from Dell et.al.
I used to work for a company targeting Fortune 500s. At that level of spend, when a client had a problem, somebody got on a plane. Only a fraction of those problems escalated all the way to R&D, which is where Oxide skills are. That's where VMWare etc are hard to beat.
Can an engineering first approach break into the cloud market? Hard to say as enterprise sales is very powerful, and the numerous "worse is better" forces always loom large in these endeavours. That said, enterprise sales driven companies are fat, slow and complacent. Oxide is lean and driven, and a handful of killer use cases and success stories is probably enough to sustain them and could be the thin end of the wedge on long-term success. We can hope anyway.
I think the question is how well they can do the management plane. Dealing with the "quirks" of a bunch of grey box supermicro stuff is always painful in one way or another. The drop shipped, pre-cabled cab setups are definitely nice but that's only a part of what Oxide is doing here. No cables and their own integrated switching sounds nice too (stuff from the big vendors like UCS is closer to this ballpark but also probably closer to the cost too).
I suspect cooling and rack density could be better in the Oxide solution too, not having to conform to the standards might afford them some possibilities (although that's just a guess, and even if they do improve there these may not be the bottlenecks for many).
Docs:
* https://docs.oxide.computer/api/guides/responses
See perhaps "This repo houses the work-in-progress Oxide Rack control plane."
On-prem buyers are doing cost reduction and cost reduction targets things like, as one example, the crazy cost of GPU servers on the CSPs. Your run of the mill stuff is very hard to cost reduce.
You can see their sort of lack of getting it by using Tofino2 as their switch. That’s just a very bad choice that was almost certainly chosen for bad reasons.
The cost case only works for GPU heavy workloads which this isn’t - wrong chassis, wrong network, etc.
Tofino2 is the wrong choice because even when they made that choice it would have been clear that it’s doa. Intel networking has not been a success center in, well, ever. That’s a selection that could only have been made for nerd reasons and not sensible business goals alignment or risk mitigation.
When you make an integrated solution you’d better be the best or close to the best at everything. This does not seem to be the best at anything. I will grant that it is elegant and largely nicer than the hyper converged story from other vendors but in practical terms this is the 2000s era rack scale VxBlock from Cisco or whatever Dell or HPE package today. Marginally better blade server is not a business.
They also make a big deal and have focused on things no one who actually builds data center pods cares about.
I actually hope they get bought by Dell or HPE or SuperMicro. Those companies could fix what’s wrong here and benefit a lot from the attention to detail and elegance on display.
My experience with the likes of Dell is that they'll deliver it but they won't support it.
Sure, there's a support contract. And they try. But while they sell a box that says Dell, the innards are a hodgepodge of stuff from other places. So when certain firmware doesn't work with something else, they actually can't help because they don't own it, they're just a reseller.
Charging for EBS local snapshots on your own Outpost S3 storage and Route 53 Resolvers on your own compute is a weird one. I don't know how they defend that. To me, it seems indefensible.
> You can purchase Outposts servers capacity for a three-year term and choose between three payment options: All Upfront, Partial Upfront, and No Upfront. … At the end of your Outposts servers term, you can either renew your subscription and keep your Outposts server(s), or return your Outposts server(s). If you do not notify AWS of your selection before the end of your term, your Outposts server(s) will be renewed on a monthly basis, at the rate of the No Upfront payment option corresponding to your Outposts server configuration.
https://aws.amazon.com/outposts/servers/pricing/
> You can purchase Outposts rack capacity for a 3-year term … either renew your subscription and keep your existing Outposts rack(s), or return your Outposts rack(s)
https://aws.amazon.com/outposts/rack/pricing/
There is no permanent purchase option.
I do wonder if AWS will eventually go down market if oxide gets any scale.
It seems like AWS has all the pieces to compete with Oxide if they care to.
Is this true? Can you set your own root of trust for the firmware signing key and build and deploy it yourself?
DC build out cost and effort
Power cooling requirements
Dark fiber bandwidth requirement
New headcount to support all this
No thanks , I have widgets to make and sell and a business to run.
And beyond that, there's all sorts of weird environments that need a lot of local compute. You'd be shocked how many servers are on a cruise ship for one example among many.
Prices are here: https://oxide.computer/sales ;)
I am not in sales and so I hesitate to speak on it in case I am incorrect, but the way that I personally think about it is that it is true that it is not an inexpensive product: there's a LOT of computer here. But the goal is to be competitively priced.
I wish there was a 4U version (10-25K but I don't think that they could come close to that price point - regardless even that is out of reach for me to ever get to play on one :-/ )
One of the most obvious examples of the problem with this approach is that they're shipping previous generation servers on Day 1. One can easily buy current generation AMD servers from a number of vendors.
They will also likely charge a significant premium over decoupled vendors that are forced to compete head-to-head for a specific role (server vendor, switch vendor, etc).
Their coupling approach will most likely leave them perpetually behind and more expensive.
But there are advantages too. Their stuff should be simpler to use and require less in-house expertise to operate well.
This is probably a reasonable trade-off for government agencies and the like, but will probably never be ideal for more savvy customers.
And I don't know how truly open source their work is but if it's truly open source, they'll most likely find themselves turned into a software company with an open core model. Other vendors that are already at scale can almost certainly assemble hardware better than they can.
It seems like they're trying to hit a middle ground between cloud vendors and fully decoupled server equipment companies.
Using Oxide is likely cheaper over the life of the hardware than using a cloud vendor. A company who already has in-house expertise on running racks of systems may be less the target market here than people who want to do cloud computing but under their own control.
True, but Oxide may find themselves competing against Dell or HP if they adopt Oxides software for their respective servers. Additionally, Oxide may find itself competing against consultants and vendors in specialized verticals (e.g. core Banking software + Oracle DB + COTS servers + Oxide software). Oxide, and their competitors are going for people who used to buy racks of Sun hardware.
> Their coupling approach will most likely leave them perpetually behind
This is a startup that took years to get their initial hardware developed. The time between this version and the version using the next version of AMD chips will be shorter than the time it took to develop this product. This is not an inherent issue with coupling vs decoupling.
Also, most servers are rarely running on the most recent cpus anyway. At least in companies I've worked at with on-site hardware they're usually years (sometimes even a decade) out of date getting the last life sucked out of them before too many internal users start complaining and they get replaced.
I'm sure they'll improve their processes over time but the lag will probably always be a non-zero value. Hopefully they'll be able to keep it low enough that it's not an important factor but as a customer it's certainly something one should consider.
It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
The talk here talks about that from about 32:15 : https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
As to your second point, unless AMD somehow becomes supply constrained and only wants to ship to their most important customers first I don't see a future where there would be any lag. Again, the delay this time is from how long it took from company start until product release. Future delays will be based on the time it takes from them getting early development parts to released products, which they could even possibly beat Dell to market on given the smaller company size and IMO more skilled employees.
> It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
I mean they've already hit tons of nasty issues, for example finding two zero-day vulnerabilities in their chosen security processor. They've shown they can work around issues pretty well.
I just think your premise is wrong - most customers don't care about not having the absolute latest and greatest. Indeed they will often avoid them because
1. They are new so more likely to have as yet undiscovered issues ( hardware or drivers ).
2. If you buy top end, they sell at a premium well above their performance premium.
ie the customers who are perennially chasing the latest hardware are in the minority.
1. The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
2. Not sure where you're getting this from. There is almost always a price:performance calculation that results in current generation smashing the previous generation with server and switch hardware. Often this means not buying the flagship chips but still the current generation.
And a major reason to get off old generations of hardware is that they become unavailable relatively quickly. It's always easier to buy current generation hardware than previous generation hardware, especially a couple years into the current generation. This has nothing to do with chasing the latest hardware.
That's not in the customers interests per se- in fact it's a pain. Having control of their own stuff could mean they could offer a much longer effective operational life.
> The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
What breaks stuff is change - sure when they are starting up it's higher risk - but again if they can manage the lifecycle better, not have change for changes sake, then they could be much more reliable.
> Not sure where you're getting this from.
I was talking about not taking the flagship stuff - which is typically a few months ahead of the best price/performance stuff.
IDK about every use case, but slightly older generations of CPUs would affect me roughly zero. I'm sure there are things so compute-intensive that one would care very much, but a lot of people probably wouldn't bat an eye about that, and not because they're unsavvy.
To the extent that these things are supported as a whole by the vendor rather than a bunch of finger pointing though, that could be massive, specifically in terms of how many staff members you could "not hire" compared to if you had to employ someone to both build and continually maintain it.
I'm posting this not to invalidate what you're saying, just to say that a little predictable upfront amount of money (the premium) will be spent very happily by lots of people who value predictability and TCO over initial price.
A moderately large Haswell cluster is equivalent in power to a moderately powerful modern server.
How is it ironic?
Also, it has little to do with the cloud; it is yet another hyperconverged infra.
Weirdly, it is attached to something very few people want: Solaris. This relates to the people behind it who still can't figure out why Linux won and Solaris didn't.
Also there are many situations where renting, for example a flat makes a lot of sense. And there are many situations where the financials and or enabled options of owning something make a lot of sense. Right now, the kind of experience you get with AWS and co. can only be rented, not bought. Some people want to buy houses instead of renting them.
Their competition has open source firmware as well:
https://www.dell.com/en-us/blog/enabling-open-embedded-syste...
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
For a mostly open source solution, not only would you need open source BMC firmware, you must have an open source UEFI/BIOS/boot firmware like CoreBoot, LinuxBoot, Oreboot, Uboot, etc.
And Canonical played with a cluster-in-a-box all the way back in 2013-2014: https://www.zdnet.com/article/canonicals-cloud-in-a-box-unde...
You could turn it into an OpenStack cloud in ~20 mins with an automated Juju OpenStack install.
Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?
If you haven't lived through this, the answer is: nobody. Everyone points fingers at the other 4 and ignore your calls.
Back in the day you could buy a fully integrated system (from CPU to hardware to OS) from Sun or SGI or HP and you had a single company to answer all the calls, so it was much better. Today you can't really get this level of integration and support anymore.
(Actually, you probably can from IBM, which is why they're still around. But I have no experience in the IBM universe.)
This is why Oxide is so exciting to me. I hope I can be in a company that becomes a customer at some point.
Dell is a single vendor that will diagnose and fix all of your hardware issues.
With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage.
I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.
And you'll be down for weeks or months while they do it.
Off by a few orders of magnitude. Dell on-site SLA with pre-purchased spares was about 6 hours.
With Oxide, you'd be lucky to get same day service.
You're talking about replacement parts. Yes Dell is good about that.
The discussion above is asking them to diagnose and fix a problem with the interaction of various hardware components (all of which come from third parties).
There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
Neither am I.
>If you hit a "zero-day" bug that Dell has never seen it's going to take time.
If you hit a "zero-day" bug that Oxide has never seen it's going to take time.
>And somehow every large customer finds bugs that Dell certification didn't.
Yes, happens. And I'm sure the exact same will happen with Oxide, so it's not a differentiator.
It's a law of computer engineering.
In the Apollo 11 decent sequence the Rendezvous Radar experienced a hardware bug[0] not uncovered during simulation. They found it later, but until then, the solution was adding a "turn off Rendezvous Radar" checklist item.
[0] The Rendezvous Radar would stop the CPU, shuttle some data into areas it could be read, and woke the CPU back up to process it. The bug caused it to supuriously do this dance just to tell it "no new data", which then caused other systems to overload.
I've been a Dell customer at a previous company. I know for a fact that's not true.
I had a support ticket for a weird firmware bug open for two years, they could never figure it out. I left that job but for all I know the case is still open many years later.
Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell. Dell is a reseller who puts components together from a bunch of vendors and it mostly works but when it doesn't, there's nobody on staff who can fix it.
I've had support tickets open for all kinda of weird firmware, hardware, etc. bugs and they've been well resolved, even if it meant Dell just replaced the part with something comparable (NIC swap).
>Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell.
Of course they do. That's like saying Oxide doesn't know how to fix stuff because they don't design the CPU, NVMe, DIMMs, etc. Oxide is still going to vendors for these things.
At Oxide, we have been deliberate at every step, designing from first principles whenever possible. (We -- unlike essentially everyone else -- did not simply iterate from a reference design.)
To make this concrete with respect to the CPU in particular, we have done our own lowest-level platform enablement software[0] -- we have no BIOS. No one -- not the hyperscalers, not the ODMs and certainly not Dell -- has done this, and even AMD didn't think we could pull it off. Why did we do it this way? Because all along our lodestar was that problem that Dell was useless to us on -- that we wanted to understand these systems from first principles, because we have felt that that is essential to deliver the product that we ourselves wanted to by.
There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
[0] https://github.com/oxidecomputer/phbl
[1] https://github.com/oxidecomputer/illumos-gate/tree/stlouis
[2] https://github.com/oxidecomputer/illumos-gate/blob/stlouis/u...
Nonetheless, it is quite interesting what you've built, but as the end user I'm not quote convinced that it matters. Sure you can claim it reduces attack vectors and such but we'll still see Dells and IBMs in the most restricted and highest security postured sites in the world. Think DoD and such. Core/libreboot with RoT will get me through compliance the same.
The software management plane y'all built is the headlining feature IMHO, not so much what happens behind the scenes that the vast majority of the time will not have a fatal catastrophic upstream effect.
>There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
That's not what I said. There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
Set aside the childish tone ...
> Dell is a single vendor that will diagnose and fix all of your hardware issues.
There are two anecdotes here disagreeing with you, and frankly that's enough to say what you said above isn't true, not universally so. I doubt Odixe is targeting big deployment like yours, but more like theirs. Whether they will succeed is another matter, but they do have a valid sales pitch and the expertise to pull it off.
Huh, then why did Sun, Oracle, and Veritas have to set up a shared tech support center in San Jose?
"Accelerated finger pointing", said a friend who had to do business with them.
At least with Oxide there is a glimmer of hope for a better future in this regard.
Yes they are using illumos/Solaris to host this but they don't sell on that, they sell on the functionality of this layer — allowing people to deploy to owned infra in a way that is similar to how they'd deploy to AWS or Azure. How much do you ever think about the system hosting your VM on those clouds? You think about your VMs, the API or web interface to deploy and configure, but not the host OS. With Oxide racks the customers are not maintaining the illumos substrate (as long as Oxide is around).
You could be right about demand, there is risk in a venture like this. But presumably the team thought about this - I think folks who worked at Sun, Oracle, Joyent, and Samsung and made SmartOS probably developed a decent sense of market demand, enough to make a convincing case to their funders.
Now you need to know both the OS they chose and the OS you chose...
(No, I don't believe it'll be 100% hands-off for the host. This is an early stage product, with a lot of custom parts, their own distributed block storage, hypervisor, and so on.)
I wouldn't have picked Opensolaris, but it's a lot better than other vendors that are either fully closed source, or thin proprietary wrappers over Linux with spotty coverage and you're not allowed to touch the underlying OS for risk of disrupting the managed product.
GNU/Linux was also "risky" at some point.
This appears to be a much better system than VMware, is free as in software, and it builds upon a free software operating system with lineage that predates Linux.
I say this in the most critical way possible, as someone who has built multiple Linux-based "cloud systems", and as a GNU/Linux distribution developer: I love it!
In hindsight of course it was remarkably prescient. This from a guy at a company that was built entirely around SGI at the time.
The first time I actually saw GNU/Linux powering something in production was in 2003, when I joined CERN and they were replacing their use of Solaris, and eventually alongside Fermilabs came up with Scientific Linux in 2004.
Later at Nokia, it took them until 2006 to consider Red-Hat Linux a serious alternative to their HP-UX infrastructure.
"Sir, if you have nothing to hide, why do you talk so fast, and pronounce words like a foreigner who learned English from a book?"
:)
We love you Bryan, never change.
I don't run on-prem clusters or clouds but know a couple people who do and, at large enough scale, it is a constant "fuck-shit-stack on top of itself" (to quote Reggie Watts). There is almost always something wrong and some people upset about it.
The promise of a fully integrated system (compute HW, network HW, all firmware/drivers written by experts using Rust wherever possible) that pays attention to optimizing all your OpEx metrics is a big deal.
It may take Oxide a couple more years to really break into the market in a big way, but if they can stick it out, they will do very well.
That's because they own the whole stack, from CPU to GUI and support it as a unit. That's the benefit of having a product where a single owner builds and supports it as a whole.
My impression of Oxide is that that's the level of single source of truth they are bringing to enterprise in-house cloud. So, I strongly doubt the innards would ever become customer-facing (unless the customer specifically wants that, being open source after all).
As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources. Their computers expensive enough to be very much business critical. You'll get some support by Oxide logging in remotely to customer systems and digging around, but pretty soon the customer will want to do that themselves to monitor/troubleshoot the problems as they happen.
Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
Apple is a great example of the benefits of an integrated system where the hardware and software are designed together. There are tons of benefits to that.
What makes Apple evil (IMO, many people disagree) is how everything is secret and proprietary and welded shut. But that doesn't take away from the benefits of an integrated hardware/software ecosystem.
Oxide is open source so it doesn't suffer from the evil aspect but benefits from the goodness of engineered integration. Or so I hope.
Afaik Android phones tend to have a lot more hardware than your average laptop, too (cell modem, gps, multiple cameras, gyro, accelerometer, light sensors, finger print readers)
I think the real benefit is being able to move/deprecate/expand at will. For example, want an app that would require special hardware? You can just add it. Want to drop support for old drivers? Just stop selling them and then drop (deprecate) the software support in the next release.
I fully agree about the evilness, and it baffles me how few people do!
Have you looked at the pedigree of many of the people behind the project? I don't say this because "these guys smart", but because these guys bent over backwards for their customers when they were Sun engineers. Bryan didn't write dtrace for nothing.
> Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
I think you're simultaneously over-estimating and under-estimating the people who will deploy this. There's a lot of companies who would want a "cloud in a box" that would happily plug hardware in and submit a support ticket if they ever find an issue, because their system engineers either don't have the time, desire, or competence (unfortunately common) to do anything more. The ones who are happy to start debugging stuff on their own would have absolutely wonderful tooling at their fingertips (dtrace) and wouldn't have any issue figuring out how to adapt to something other than Linux (hell, I've been running TrueNAS for the better part of a decade and being on a *BSD has never bothered me).
As we see nowadays on tablets and laptops, most OEMs are quite keen in returning back to those days, as otherwise there is hardly any money left on PC components.
Or Linux running underneath all the Java-y Android stuff.
Also, Apple is just the one who survived. Previously I'd have thought of SGI, DEC, Sun, HP, IBM, Dell some of whom survived some not.
Those three consumer products I mentioned each provide a platform for a user and business space to floroush and thrive. I expect a company doing something similar for cloud computing to want the same. But it will require some magick: momentum, money, trust. That kind of stuff, and loads of it. (With some big names behind it and a lot of FOSS they got me excited, but I don't matter.)
Their choice in foundation OS (for lack of a better term) really should not matter to any customer.
Now imagine a multi-million dollar mission critical pile of computers running on premises, and your sysadmin being able to do so.
Oxide is closer to a rack of Supermicros than AWS.
Solaris is still Solaris, as of the latest release last month. OpenSolaris hasn't been OpenSolaris in a while and is Illumos, yes.
Could be! Seems too early to tell though, and remains to be seen whether it pencils. Which is the whole idea of starting a new venture, no?
One of the more interesting discussions I had during my tenure at Google was about the "size" of the unit of clusters. If you toured Google you got the whole "millions of cheap replaceable computers" mantra. Sitting in Building 42 was a "rack" which had cheap PC motherboards on "pizza dishes" without all that superfluous sheet metal. Bunches of these in a rack and a festoon of network cables. What are the "first class" elements of these machines? Compute? Networking? Storage? Did you replace components? Or a whole "pizza slice" (which Google called an 'index' at the time). Really a great systems analysis problem.
FWIW I'm more of a "chunk" guy (which is the direction 0xide went) and less of a "cluster" guy (which is the way Google organized their infrastructure). A lot of people associated with 0xide are folks I worked with at Sun in the early days and during that period the first hints of "beowulf" clusters vs "super computers", was memory one thing (UMA) or did it vary from place to place (NUMA). I have a paper I wrote from that time about "compute viscosity" where the effective compute rate (which at the time largely focused on transactional databases) scaled up with resource (more memory more transactions/sec for example) and scaled down with viscosity (higher latencies to get to state meant fewer transactions/sec) Sun was invested heavily in the TPC-C benchmarks at the time but they were just one load pattern one could optimize for.
These guys have capitalized on all that history and it is fucking amazing! I just hope they don't get killed by acquisition[1].
[1] KbA is a technique where people who are invested in the status quo and have resources available use those resources to force the investors in a disruptive technology to sell to them and then they quietly bury the disruptive technology.
Is the unit of composition a rack (chunk), a server (smaller chunk), or a blade (smallest chunk)? In what I think of as classic systems architecture you've got a 'store' (storage), 'state' (memory), 'threads' (computation), and 'interconnect' (fabrics). In the 90's a lot of folks focused on fabrics (Cray, Connection Machine, Sun, etc) somewhat on threads (compute blades), and state came along for the ride. How these systems were composed was always a big thing, then along came the first Beowulf clusters that used off the shelf motherboards (a "chunk" of threads/state/store) with a generic fabric (Ethernet). Originally NASA showed that you could do some highly parallel processing on these sorts of systems and Larry and Sergei at Stanford applied it to the process of internet search.
Collectively you have a 'system resource' and with software you can make it look like anything you want. When you do compute with it, its performance becomes a function of its systems balance and the demands of the workload. Its all computer sciencey and yes there is a calculus to it. This isn't something that most people dive into (or are even interested in[1]) but it was one of the things that captured my imagination early on as an engineer. I was consumed with questions like what was the difference between a microprocessor, a mini-computer, a workstation, and a mainframe? Why do they each exist? What does one do that they other can't? Things like that.
[1] At Google I worked in what they called 'Platforms' early on and clearly most of the company didn't really care about the ins and outs of the systems bigtable/gfs/spanner/etc ran on, they just wanted APIs to call. But they also didn't care about utilization or costs. By the time I left some folks had just figured out (and one guy was building his career on) the fact that utilization directly affected operational costs. They still hadn't started thinking about non-uniform rack configurations for different workloads.
Private equity in the US has collectively determined that no company shall exist outside of investment ownership. I don't know what the ownership structure looks like, but generally speaking, it seems that nearly everyone has a "fuck you" number. Now that Oxide is venturing into Dell and HP's turf, I worry someone will get a fix on what Brian's number is.
>They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration >problem. They integrated it.
that's only true if you think that "users" means "people who operate cloud computers", which is about as far from understanding what Stallman is talking about as is possible. Someone who makes SaaS and runs it on an Oxide computer is no less of a rentier capitalist than someone who makes SaaS and runs it on AWS.
As a technologist, I really appreciate what they have done. Impressive work, high quality, however I don't understand who this is for.
The meaningful market for Data Center hardware is pretty well defined in two clusters. People that build/make custom gear (such as Hyperscalers) and people that buys HP/Cisco/IBM/Dell... (blades or hyper-converged). To scale, you obviously want your DCs as standardized as possible.
Until this company has a certain/size and scale, no one serious will trust their black boxes at any type of scale.
Beyond the tech, how would support services really work? We can have a technician from any of the large vendors on-site in less than 2 hours. In some of our DC clusters we actually have vendor support personnel 24x7 on-site with vendor paid spare parts inventory. How would they provide that level of service?
Maybe I am not the target audience for this offering.
I think the motivations for why people would want to own their own are probably a mix of financial (at a certain scale there's a tipping point and it gets cheaper), and regulatory/compliance/whatever, like if it's healthcare data, or defense, etc.
I do understand well the rational of running your own servers vs hyperscalers, as well as the repatriation trend but I see Oxide at best as a niche player.
They pay other people to do that and they don't really care if it's a miserable time. And if it takes days instead of hours, who cares? Rarely is someone setting up a data center under the gun (unless you're Elmo and we all saw how that went).
Factors like scalability and ongoing support are much more top of mind.
Not saying that Oxide can't address this, and I love Oxide's focus on the experience, but I think this bottom-up approach to convincing customers is going to be a steep climb..
But they seem to be up for steep climbs, so I wish them all the best!
Why “Elmo”? https://www.businessinsider.com/twitter-insiders-users-calli...
data:text/plain;base64,aHR0cHM6Ly92ay5jb20vd2FsbC00ODQxMzg3Ml82NjAyNg==
I mean... it went quite well all considering? There was some site instability for a brief period of time and now it's back to working normally. The initial hypothesis was that the data center was vital and so couldn't be shut down quickly. Turns out the hypothesis was incorrect. So I'm not quire sure that makes the point you're trying to make.
Of course Twitter is definitely not all use cases, so trying to generalize from one data point isn't a great idea in general.
Elmo's experiment came to mind so I tossed it in there.
I'm not sure how big that space is, or whether it is likely to grow or shrink over time, but I'm intrigued by the proposition they're testing here!
I know there's already plenty of concerns about giving cloud-based AI services access to corporate data, so maybe there's a growing market there..
HPE Simplivity has done well.
Also, Nutanix.
Oxide is entirely integrated with their own custom (though based on OSS?) networking via Tofino based switches and software. Could be good, but a lot of subtle bugs can occur at this layer. It's a risk.
I set up some Cisco server hardware a few years ago, and only by the time I'd managed to order it I was already wishing I had a better choice. When it arrived and the remote serial was unusable to fix the BIOS ("American Megatrends copyright 1984" at 9600 baud? No thanks.) I was ready to give up and go back to AWS.
This is a market ready for a kick in the ass, which Oxide plans to do.
All Oxide has to do to win that market is ship software and firmware that doesn't suck, because there are incumbents but the incumbents are clearly incapable of doing so.
But precious few new ventures have ever entered a market where nobody said some form of "I don't get it, there are already established products in this space, how will this new thing ever get traction?". And yet, lots of new ventures turn out to be competitive with the established players in their market, for one reason and another.
Yea, there's definitely a market in defense here. Because even though Azure/AWS offer Govcloud, its inadequate for non-civilian connected infrastructure. This offers benefits of writing "modern software" and deploying it in similar modern fashion while keeping it completely running isolated. Imagine being able to make your command and control operations actually decentralized and not vulnerable to a missile strike on a single datacenter.
How many times has computing hardware changed in response to the economics of the parts and the economics of the businesses buying hardware?
There are downsides to new models, but money solves a lot of problems
So I don’t know about Oxide in particular, but it seems short sighted to bet on stagnation
Also Oxide is doing what Google did 20 years ago, and Facebook open sourced ~10 years ago, so it’s not exactly unproven
I'd be curious to see what companies are interested, Oxide doesn't have any logos on their website which is a little odd given the space.
If you're working for a megacorp nothing tends to happen quickly, so there will be a slow roll-out over a multi-year period as old hardware gets phased out and new hardware is brought in for a refresh.
If there's a hiccup at any point they'll simply keep the previously purchased stuff running and start a new roll-out with another vendor next fiscal.
> I'd be curious to see what companies are interested, Oxide doesn't have any logos on their website which is a little odd given the space.
1. They're just starting out. 2. Some of their customers want to be (or start-out initially) discreet:
> Oxide customers include the Idaho National Laboratory as well as a global financial services organization. Additional installments at Fortune 1000 enterprises will be completed in the coming months.
* https://www.bloomberg.com/press-releases/2023-10-26/oxide-un...
I bet the trick is to keep the perfect balance of high profile wins and and losses. You want to be defensible as an expert to the non-technical folks while obviously a fall guy to the technical ones I guess.
I think these guys aren’t that, though, they seem to be selling a real, cool product.
That equation changes with the entire software stack being open-source
This is the secret none of those existing vendors (Dell, Lenovo, HP, etc) are willing to tell you: They have very limited technical expertise on what they sell you and outside of some specialized troubleshooting they can do, they'll defer to their vendors. The understanding is that you've got the intellectual horsepower on staff to cope with their various shortcomings.
Of course that framing itself is bad - F100 companies aren't usually quite that monolithic. By the time they get that big there's a heterogeneous set of processes, equipment, systems, etc. Some parts of the company may use Oxide right away because they see it as a solution, and others may keep using the IBM mainframes, and other still will keep using racks/blade servers from Dell for eternity.
Any hardware contracts are very long term and you'll have a hard time getting me to switch to a different vendor, especially when they also want to come in with an unknown operating system which I have to run.
With compute one size doesn't fit all, maybe you need more disk space or maybe you need GPUs... I'm sure Oxide will come out with different spec modules over time.
The idea is similar: It's a rack which runs virtualized workloads and you don't have to think much about individual machines.
- Why would anyone buy the Framework laptop, they don't have nearly the support/pedigree that Dell, HP, etc. has?
- Why would anyone use iPhones in the enterprise/IT world, they don't have nearly the support/pedigree that Blackberry, Microsoft, etc. has?
- Why would anyone use Google Fiber, they don't have nearly the network or support that AT&T, Spectrum, etc. has?
- Why would anyone ever use Linux (in enterprise, let's say), compared to the support and adoption that Microsoft/Windows offers?
- ...
I'm purposely picking different examples with varying degrees of success or adoption. I am not claiming that Oxide will be an instant category-dominating success. I don't think Oxide expects to replace HP/Cisco/Dell/etc. overnight, and I don't think a business has to launch with that ambition from the start, to prove that it's worth launching.
But this take is so repetitive as to be bordering on cliche -- I don't know if you're self-aware enough to realize, you are literally just a living embodiment of the "Innovator's Dilemma" right now...
Framework is niche.
iPhones do have the pedigree.
Google Fiber is barely used.
Most folks do use a supported Linux distribution, they don’t roll their own.
Not in 2007-2008 which is equivalent to Oxide today.
Not to mention the comparison is inane to begin with. Using an iPhone for your enterprise and moving your tech infra to a relatively unknown company are not equivalent at all.
This just seems like a twist on hyper-converged infrastructure + open source.
I mean, I love their design. I just don’t think it’s special enough to warrant mass adoption. And I certainly wouldn’t be deploying at scale in an enterprise environment with zero widespread adoption.
Small entities will just use cloud same as always. Large companies have a multitude of unique needs that won’t all be catered for by a single box. The big vendors will clean up on that front as usual.
"Niche" is not a dirty word. Framework is niche, but if they can make their cost structure work (which I have no idea whether or not they can), that's ok. New products don't need to dominate their market in order to be succesful.
(But I do agree that Google Fiber is a bad example / a good example of a new product in an established market not working.)
"Hey subdivision A, could we buy a few Oxide racks and move your workload there from AWS? It looks like they would have all the storage and compute you need. Yes? Ok, in 36 months we'll pay your current IT department employees a bonus of 50% of whatever it has saved us vs your current AWS budget."
Like which?
Or in other words, is migrating to Oxide somehow assumed to be easier than migrating to some other non-locked-in cloud infrastructure?
But the point still stands. There's a lot of AWS spend happening (even after being optimized) that is frustrating when you look at the raw numbers and consider how much server capability you could outright buy for the same amount. And Oxide would make it so much easier to run a bunch of VMs (and infrastructure-as-code) than standard racked x86 servers.
Oxide appears to be a complete shoo-in for companies that used to run a bunch of VMs on racked Dell/HP servers, migrated their VMs/storage to AWS, hate their monthly AWS bills, and still have the old server rooms available.
Since many of those use cases probably already run extensive on-prem infrastructure this could appeal to them. AWS outpost talks about industries like healthcare, telecom, media and entertainment, manufacturing, or highly regulated spaces like financial services. I've heard of media companies that process through things like IMAX cameras that have just tons of TB's of data sometimes just for 5 minutes worth of footage. That would simply be too cost prohibitive - in bandwidth alone - to try and move around in the cloud and you don't want to have to wait for things like AWS snowball or whatever.
While I think the space is "niche" those niche spaces are not small. Big companies with big budgets.
Their marketing and story is supposed to convince you that you could save money running their things rather then Dell. And instead of paying for VMWare you get Open Source Software for most of it.
> Until this company has a certain/size and scale, no one serious will trust their black boxes at any type of scale.
I guess that a risk they are willing to take. Some costumers might wait for a few years until they see Oxide being big enough.
Other costumers might be sick of HP/Dell and might take a Risk on a smaller company.
Since they seem to have some costumers, some organizations are willing to take the risk to get away from Dell and friends.
So I think you are the target audience but you are not willing to risk it until they are larger and less likely to fail and they have a good story in regards to support. I assume they have a support story of some kind, no idea what it is 'Contact Sales' ....
In terms of 'trusting they will continue exists' all they can do is survive for a few years until they are pretty established, then more people will be willing buy their product. And hopefully in that time their existing costumers rave about how amazing the product is.
Lets hope they don't go bust because all potential costumers are just waiting. Then again, you can't anybody for not buying from a startup.
As someone who has dealt with mostly Debian and Ubuntu in recent years, every time I had to deal with even small numbers of RHEL licenses I often asked myself "Why do put up with this?" (I know why, but still… such overhead.)
That's mostly speculation but it seems like it has to be true.
Unrelated, but a "costumer" makes costumes. I think you were looking for "customer".
Forgive me for being naive, but that sounds like a great way to differentiate their offering.
E.g. the famous Maytag Repairman, who sits at his desk doing nothing because Maytag washers are so reliable that he has nothing to do.
> In a time in which the laundry appliances of major manufacturers had reached maturity, differing mostly in minor details, the campaign was designed to remind consumers of the perceived added value in Maytag products derived from the brand's reputation for dependability.
One thing that Bryan understands is that you can "lock" the customer in with great products and services, as well as continuing development, while also making the customer feel secure in having a way out should you turn into a company that treats locked-in customers as cash cows. The open source strategy (it is a strategy for Bryan and Oxide) is there precisely to do this: make the customer feel they can leave you, but then not.
For your deeply technical staff, having source code access is a big deal too, since it enables them to better understand the products they use.
How big is the market of sufficiently-vendor-lock-in-averse customers? I don't know -- that's not my remit. But there's the size of that market right now, and whether Oxide (and any other companies with similar visions) can grow that market by sheer willpower. I make no predictions.
What if Oxide can get the next Netflix to use their stuff instead of a public cloud?
One of the amazing shifts in the last 20 years was realizing that commodity hardware, when deployed correctly, could do the job.
If you're buying millions of this stuff, what says that you're going to get support for it in 5 years. Who knows... maybe Cisco wakes up and gives them an offer they can't refuse and then shuts down the company.
By the way, people endlessly gripe about Google deprecating things and that's just software...
My two cents is that this isn't really an innovation business model, it's an execution business model. Their proposition seems to be that they can execute a a server rack with integrated hardware and open-source software so well that customers will love their product.
Maybe they'll just end up with tons of nerds our there thinking, "I really wish I could justify investing in Oxide racks, but it just doesn't make sense for my business / I just can't sell it to my CIO", but hey, maybe not!
It's better to just think to yourself: "this doesn't seem like a useful product to me, I don't understand why people find it interesting, oh well", and move on.
I actually do think what they are doing is innovative across the board. They are taking all of the common feedback anyone who has run large scale data centers (which I have) and applying it to a brand new product. Unfortunately, they are doing it in a way that is extremely vendor specific.
`import { Api } from "@oxide/api"`
`Understand and debug issues faster`
They've got 231 repos in github...
Metaphorical fans, but also fans, and everything else. It was an example from the blog post. I looked at that super innovative and cool backplane. Sure, warranty covers it for the first N years, but then what? What happens when things turn into a Tesla situation and you get horror stories of delays and poor results?
I think 'commodity' is generalized at this point. Given the choice of Supermicro and Oxide. I'm going to pick Supermicro. They can deliver me a clean rack of machines too. Why would I go with Supermicro? Big company, lots of products, lots of choices, I can work with them to get what I want.
Oxide is too singular. It is one rack, one design, one set of specifications. Don't get me wrong, there is some value in that... and I'm sure you'll sell plenty of hardware, but I'm not finding the value in it for myself. That's the part that I mean by 'commodity'.
As far as I can tell, there exists no "open source stuff developed by a large community of people" to run integrated server rack hardware. But if you're using such a thing, then I'd like to know what it is!
Very. Just look at the USA defense spending budget. If you’ve ever worked on AWS-govcloud or secret, you know there’s a market here.
This has huge use for military too. Imagine having a black site or off-grid location but still needing a rack of things. What if you could spin up an entire enterprise infrastructure by just loading up this rack?
If this team manages to get this thing government certified, there’s a lot of profit to be had.
This is already being done quite successfully with existing offerings. Flexpod, VxRack, Nutanix, there are a ton of options.
"Other products already exist" is never a strong counterargument to the existence of new products.
https://web.archive.org/web/20080705140230/http://blogs.sun....
And "FYO" stands for Fuck You Oracle.
My guess, someone who buys hyper-converged infrastructure today (e.g. Nutanix, VCE, etc) ... but that market is getting smaller and smaller by the day.
I expect the oxide supporters to have a hard few years ahead of them of finding bugs in high throughput environments. But at the end of the day it will be worth it just to have another competitor in a pretty boring playing field.
I can't think of a lot of examples right now but I can already imagine one type of customers for such a product - universities wanting high performance computing. At my alma mater, the HPC cluster/server was in a different slightly distant location. Using something like AWS wouldn't be liked by almost any uni admin, and running a server on premises isn't a great idea in a place that gets the occasional (but rare) power or internet cut. Outsourcing some of these responsibilities may have been nice for our admin.
This gets to the crux of my first thoughts when I read the marketing copy: can they deliver (reliability).
They do admit very clearly that what they're doing is hard and that at many points during development they were reluctant to be too ambitious (for obvious reasons), but at each stage they did just that: proceeded with the most ambitious option. That takes a huge amount of self belief that might be warranted or might be hubris. As you point out, untimely the success of Oxide won't only hinge on ability to deliver on that self belief, but more on their ability to convince prospective customers that they have competency to do what noone else can.
I fully support every part of their approach in theory but my wizened traveled self thinks it smells a little too good to be true.
A lot of people didn’t understand the need for the iPad mini or Mac mini or myriad other products without an obviously existing market. I think this is extremely keen of Oxide to fine the borders between these offerings and fit very snugly between them.
The current state of the art is a fucking train wreck. Choose any layer of abstraction and start picking at it and you’ll find mostly gaffer tape. Scaling up sucks. Rewriting old monoliths to micro services wasn’t a panacea except maybe for cloud vendor profits. You said the hardware market is well defined, which is another way of saying ossified. Particularly when you start comparing it to the expected pace of software. How’s Open19 going? How much liquid cooling do you have in customer racks?
In my opinion this is ultimately a software offering that happens to come on vertically integrated hardware. It offers a complete, highly polished API to a minuscule-scale DC. If they can find market fit and make a little bit of money, the next step is to start making deep improvements to things behind the abstraction. From where you sit, you are well aware that you could make huge improvements inside the DC demarc if you could do it without disrupting customers. But you’re probably limited by the terrible, terrible APIs you would be forced to use, that don’t offer the capabilities you need, from vendors who would be happy to chat but would provide timeframes in the 6-18 month range for even a modest improvement.
So this, IMO, is about defining that interface to a DC on a single hardware platform with vertical integration, and then scaling up from there. The current hyperscalers will adopt the suite of APIs and capabilities directly, make lower quality copies, or die.
Of course all assuming they’re successful enough to get the revenue machine going in the first place which seems likely to me, given the absolute dogshit state of the cloud world today, the trend towards multi cloud, and the business case for moving certain loads back on prem or to a bare metal colo++ offering.
The problem is that the industry hasn’t noticed this yet. The hyperscalers have, and are printing money as a consequence.
Congrats to the team on reaching this big milestone and (in my eyes at least) just as much congratulations on doing it in a way that has been unique and sticking to values that seems to drive a strong positive culture (at least from the outside looking in).
Shipping products is hard, and only getting harder. IMHO, one of the big drivers of that is just how complex every market has became. Building and selling software alone is so much more multi-disciplinary than it was 10 years ago and adding hardware to the mix is upping that by a huge factor. As I look around, I see so many companies struggle to build teams that can handle the huge range of required tasks. To see a company like Oxide that (once again, from the outside a least) seems to have things together on so many fronts, especially while doing it while sticking strongly to some core values, is pretty inspiring.
Not to get overly cynical, but I don't think it is an extreme opinion to say that current start-up culture feels like you have to make big compromises in what you believe in order to be successful. Whether that be open-source, how you value and pay employees, or even just rushing things to deliver that aren't ready.
While I acknowledge Oxide has some well-connected, experienced founders that I am certain enabled them to get the resources and trust to do things their way, I really hope they kill it so that other founders and builders can learn that you still can build not just financially successful products, but great organizations that truly care about their values.
And shipping full cabs is harder!
I've worked in startups most of my life, but it's tough feeling like you've got to take a bunch of shortcuts ("MVP"-it before you run out of runway, etc. etc.), so to see a startup just put a ton of real engineering prowess into a great product is a sight to behold.
Also, while there are obviously amazing products out there in the world, it's so often a "race to the bottom" that it feels like decently made products are just prohibitively expensive. I'm really into antique espresso machines, and there is a reason that seeing rebuilds of beautiful old lever espresso machines get so many oohs and ahhs online - they truly don't make them like they used to, so it's so cool to see these beasts brought back to life. For some reason I have a similar feeling looking at these Oxide racks - they just didn't cut corners to build an awesome machine. Again, not qualified to say whether that makes economic sense long term, but it's still a thing of beauty regardless.
Semi-sort-of related... there was an automation company, Bedrock automation, that went defunct about a year ago. Their PLC hardware ideas were dope but I always felt like they were missing the boat a little bit by supporting stale PLC programming languages. I used to wonder if supporting Rust and Ada on these PLC's would be been a good idea to diversify/specialize into complex control system domains. Also, iirc, Bedrock didn't support EtherCAT which I felt was a mistake.
Anyhow... one of these days it would be cool for a forward thinking company like Oxide to tackle what a modern complex, distributed control system hardware/software stack should look like using Rust/Hubris.
Operation/Shop floor technologies (OT) are treated like mechanical equipment, "When it fails, we'll swap it out for a new one." Well, this isn't a motor, it has programming, and it interfaces with I/O sensors and devices.
The main challenge is the lack of knowledge and skills in modern technologies among technical staff and decision makers in industrial organizations.
A an aside, in my early days I hopped on a 5-hour flight and drove 6 hours to replace a failed hard drive in a Windows NT machine used as an HMI. Then a year later, replaced it and all the other clients with a vSphere stack. The local resources, both internal and external, were too intimidated to touch it.
I'd be in favor of a "reset" in automation, it feels like fighting city hall.
Beckhoff & B&R Automation are part of the way there. They both have some really slick portfolios. IMO, on paper, these are the two most advanced control system companies (with large sales volume).
It's a critical issue. Resources are wasted, and soon we won't be able to waste as much as we did, as the abundance era is coming to an end. The desire and need for higher efficiency through highly complex technology is only to increase, as the population gets older and older.
I can easily see mandatory IT training (like the current mandatory army training in some countries) and states standardizing certain technologies to avoid fragmentation and thus freezing progress, for good reasons. Like forbidding any database to be used in production except 3 most complete ones or banning all web frameworks except React and Svelte (arbitrary). It's not that they won't want progress, but they won't be able to afford progress and the resulting skill fragmentation in the IT workers while people are hungry and money is ever tighter.
“Stale PLC programming languages” might be stale and in need of a rework, but “rewrite in rust” entirely missed the value proposition. Memory safety is important, sure, but these systems aren’t usually ever allocating memory or taking actions that aren’t deterministic. PLCs usually operate in strict realtime environments. Like each operation contains a known number of clock cycles, and is timed to match physical “stuff” moving in the real world. To rewrite in rust, you’d need a flavor of rust where you can deterministically know the timing of ops and calibrate the timing of each branch of any control paths.
That said, there is a market for softer realtime PLCs and companies like Rockwell sell Windows IoT hardware- but they still contain a strict-realtime companion controller. Arduino is now selling a mountable PLC that can be programmed using their tools, but even they support traditional PLC programming languages.
Rust and C code can also operate in strict real-time embedded environments provided some basic rules are followed like "no dynamic memory allocation". This done all the time.
If one follows hard real-time coding standards like Misra C or "JPL real-time C" (and these standards can also be applied to Rust, Zig, Ada, etc.) the code will run deterministically on a given target... just like PLC code running is not necessarily any more deterministic than C code (no dynamic allocation) running on an embedded processor. In fact, many PLC's today run a "PLC code execution" engine on top of a real-time operating system like QNX (which I think is mostly C code). Even in some older PLC's, it is still C firmware that's interpreting and running the PLC code.
PLC programming "languages" (ladder logic, FBD, CFC) were designed for their programming audience, who are not software engineers. It is difficult to represent complex logic and numerical code in these languages and that limits the sophistication of algorithms that can be implemented on these systems. For instance: try writing a model predictive control routine in ladder logic; I think it could be done but I'd lose my remaining hair doing it.
PLC Structured Text is very similar to Pascal and pretty capable, but folks generally don't write model predictive control algorithms in Structured Text either.
It is hard to orchestrate multiple PLC to run as a cohesive, deterministic unit. The interfaces between PLC often need to be kept simple and sometime this communication is less real-time. Things are a lot more advanced in cluster computing.
The limitations of current PLC architectures is already a pain point for complex control system, like large robotic manufacturing lines or optimal control of HVAC in large buildings)... and it's going to get worse as performance demands increase. Again, as mentioned above, I think Beckhoff and B&R are further along the evolutionary path than others in the industry.
Spec wise, some low power systems like an Intel NUC, LattePanda Sigma, or Zimaboard. You could fit 3/4 of them in a single 1u with a shared power supply. They could even offer a full 1u with desktop grade chips on the same sleds.
I have thought about building one myself, but it's a large investment of time that I can't seem to find lately.
Price point?
I work with some businesses who need very, very reliable, high-bandwidth, and low latency connectivity to their data. The amortized cost of on-prem beats the cost of any off-prem offering as soon as the cost of the necessary connectivity is factoted-in.
Your not going to find any serious hardware product with reliability guarantees, in writing, for much less than half a million anyways.
That said, where this product-space gets tough is actually scaling it down. It's pretty challenging to create something that is remotely stable/functional in a homelab (space/power/money) budget. Three servers and a switch would probably be the bare minimum. We (and I'm sure Oxide :) scale up like a dream.
A lot of homelabbers (and even some small businesses) go for Proxmox as a virtualization distribution. I don't use it myself, but IIUC it's effectively a Debian distro packaged to run KVM/LXC, with support for things like ZFS, Ceph, etc. It has some form of HA, an API used by standard open source devops tools, handles live migration, etc.
So buy some used rack-servers on Ebay (or new, if you're ballin'). A lot of businesses sell their old stuff, so you can pick up a generation or two out of date for a good price. If you want to do fancy stuff like K8s, Ceph, etc you'll probably want at least three nodes, ideally more, and a bunch of disks in them. Networking gear is a sort of pick your poison thing. A lot of people love Ubiquiti gear; a lot of people hate it. TP-Link is another that's good and budget friendly. StarTech sells smallish racks (including on Amazon), if you want to start there.
It won't look exactly like SoftIron's HyperCloud or Oxide's Cloud Computer, but you can certainly get pretty sophisticated.
Not sure if this answers your question, but other great spaces to explore are the 2.5 Admins and Self-Hosted podcasts.
To go ahead and dream a bit:
I'd hope for an online configurator like the one SoftIron's HyperCloud has [1] but instead of "talk to a sales rep", show a price for what you just configured, like you're configuring a macbook.
Relatedly, there should be a standard rack form factor in the size category of NUCs and Mac Minis, rather than having to go all the way to the 19 inch monster racks that medium to large businesses use. If it were nailed down to the point of being able to blind mate (just learned that term from Oxide's article here!) gear into it, that would be kind of perfect.
1. https://softiron.com/hypercloud/configure/I’ve not personally used it, but their stack of software is open source, and according to some commenters in the thread, super high quality.
https://canonical.com/blog/jumpstart-training-with-the-orang...
https://arstechnica.com/information-technology/2014/06/hands...
Don’t know if oxide would want or be able to compete in the low cost market but a bigger a more expensive desktop/workstation as a mini homelab cloud could be a great option to get people trained on the oxide platform.
If you want to play around with their Hubris OS: "You wanna buy an STM32H753 eval board. You can download Hubris, and then you’ve got – you’ve got an Oxide computer. You have it for 20 bucks.”
Do they actually assemble and build their own racks of hardware, or is that outsourced? Somewhere, there must be an assembly plant. If this stuff actually exists. It's hard to even find pictures of their products. Do they have production installations?
First rack shipped to costumer Jul 1, 2023: https://twitter.com/oxidecomputer/status/1674901883130114048
Here is a picture an incomplete rack: https://pbs.twimg.com/media/FfT7MHoUoAE90QZ?format=jpg&name=...
You can find other pictures on twitter and other places.
Certainly odd but not out of the ordinary for a small Bay Area start-up where the Founders have a ton of cash and the focus is mostly on what they personally want to do. Posts like these are b/c the Founders want to hire 1-3 people who fit exactly into alignment with them.
We’re about 60 people, currently.
The most recent few episodes have been about corporate open source and they've had excellent guests, like Kelsey Hightower. Definitely the best computer related podcasts out there. Bryan and Adam are great hosts and their humor is always a delight.
Other podcasts I'd recommend: ADSP [1] (if you're into programming), 2.5 admins [2] (if you're into computers). But I have no recommendations about hardware design because AFAIK the podcasts you mention and what Oxide is doing are pretty unique.
[1] Algorithms + Data Structures = Programs https://adspthepodcast.com/
[2] Allan Jude, Jim Salter, Joe Ressington https://2.5admins.com/
The shows at https://www.jupiterbroadcasting.com/ are my personal go-to.
Such as?
I’d say their opinion on “ransomware being a backup problem not a security problem” was enlightening and their backup opinions are also interesting (Push vs. Pull, ZFSes snapshotting, etc.)
Allan’s experience in FreeBSD is welcome too, not every day you get to hear about FreeBSD in prod.
They both skew FOSS too but not in an idealistic way, it’s based on decades of experience.
In a similar vein, Embedded.fm is a great podcast for embedded SW. Though I bet Oxide's take on embedded rust is a lot different than the hosts of that show!
https://softiron.com/hypercloud/
https://softiron.com/blog/run-bmc-why-we-decided-to-build-ou...
I believe our smallest configuration is 5 servers and a few switches.
Our API is also under active development, but it is open source today:
https://github.com/SoftIron/hypercloud-api
Go bindings included, other languages are under discussion.
And the debugger is here: https://github.com/oxidecomputer/humility/blob/master/cmd/ri...
I also gave a talk at this year's Open Source Firmware Conference that covers a bunch of debug strategies: https://www.osfc.io/2023/talks/unplugging-the-debugger-live-...
(conference videos aren't online yet, but should be posted early next week)
I design them in Monodraw, pass it through a janky converter I wrote that converts text into a json grid of characters. I then render a number of layers that get combined, which is a mix of the static art layer, and others generated from functions that spit out a similar cell based frame.
If you're interested: https://gist.github.com/benjaminleonard/c913ddbf23fe7a70f9c2...
And for what it's worth there's this ASCII game: https://twitter.com/StoneStoryRPG
So, Humility is like MC/DC?
https://en.wikipedia.org/wiki/Modified_condition/decision_co...
1. The state or quality of being humble; freedom from pride and arrogance; lowliness of mind; a modest estimate of one's own worth; a sense of one's own unworthiness through imperfection and sinfulness; self-abasement; humbleness.
-- Webster's 1918 Dictionary
Our avionics software was originally spec-ed to not NEED a debugger it was to be of such high quality, but the designer added break point op-codes in the VM anyway. I guess he was aware of the concept of humility, too. ;) Like humor, humility seems to be in short supply these days.
Hmm...maybe I should have referenced:
https://en.wikipedia.org/wiki/Sacred_cow_(idiom)
> The motto of the satirical magazine The Realist was "Irreverence is our only sacred cow".
Humor does seem to exist today, it just has a narrow band-pass filter, with apparent focus on "organic" product marketing.
When I see the concept of "extremely high quality" applied to software in the aerospace domain, I tend to think of methods like formal proofs and "semi-formal" methodologies like DO-178C (including MC/DC, not as a tool, but as a critical sub-process). It does appear that Humility adheres to some traditional ideas used in even "safety conscious", real-time embedded work:
> However, Hubris may be more interesting for what it doesn't have. There are no operations for creating or destroying tasks at runtime, no dynamic resource allocation, no driver code running in privileged mode, and no C code in the system. This removes, by construction, a lot of the attack surface normally present in similar systems. [1]
Oxide surely didn't invent all these concepts, we were applying some of this in the early '90s on human-safety-critical embedded projects, and they were most likely used before that. The attack surface eliminated is more than just security, e.g. even self-induced "attacks" from task priority inversion. Maybe the concept of applying them to rack mount servers is novel, but I have no experience there. Techniques like DO-178C go farther and do things like trace requirements to object code and strict coverage criteria.
When you are considering where Hubris/Humility might be applied in aerospace, also note that it would most likely be denied at the proposal stage of talking to a regulatory body like the FAA if you said, "The methodology for our brake controller is cloning a Github repo written by 100X S/W engineers in the Rust language, then modding that to fit." It's a process you have to follow in a documented fashion, from the start. Who knows--I certainly don't know everything--if you do have a identify reproducible process that improves S/W quality, the FAA might be interested.
And as far as eliminating debuggers goes, Boeing did actually do a joint academic/industry project on a zero-bug reduced (subset) Ada compiler (Zbra), which itself would be qualified as a tool instead of the more laborious process of mapping requirements to object code directly. [2]
In the spirit of humor shown on this topic, I would suggest considering a reduced subset of Rust called Iron. :) Regards, and best of luck!
[1] https://oxidecomputer.github.io/hubris/
[2] http://www.sigada.org/conf/sigada2002/SIGAda2002-CDROM/SIGAd...
* Memory access is protected, but addresses are not virtually mapped. The lack of paging of course ties strongly to the inability to dynamically add tasks.
* They wish they could have read-only shared libraries (viable since there's only one address space) so they could truly be shared, but the current ecosystem assumes mutability is possible (even though in Rust mutable globals are rare).
Can't wait to read its source code, I was curious since reading this thread[2].
[1] https://github.com/oxidecomputer/helios (if this is not found for you, we're in the same boat)
I would expect nothing less from that crew. They did amazing things at Sun Microsystems, Inc. (RIP), and they continue to do even greater things now.
https://github.com/oxidecomputer/hubris
And, that one underscore-delimited folder name in this repo just catches the eye, huh?
With respect to Humility, I am going to resist the temptation of pointing out why one of those directories has a different nomenclature with respect to its delimiter -- and just leave it at this: if you really want to find some filthy code in Humility, you can do much, much better than that!
[0] https://github.com/oxidecomputer/hubris/commit/651a9546b20ce...
Add few add ons, service contract, support etc so would be there.
I want such a rack but power draw on average is listed around 12kwh. Unbelievable.
Or provide enough white glove after-sales support and written guarantees to peel away low end mainframe customers at a fraction of the price.
Given their management plane/API:
* https://docs.oxide.computer/api/guides/responses
the performance may be about the same, and CapEx as well (or maybe a little higher), but OpEx could be where you make it up in large(r)-scale operations.
And space efficiency is also not to be sneezed at: for some operations DCs/compute can be place anywhere because latency is that big of a deal, but in other places you need to be close to certain things (trading), and real estate can get expensive.
https://oxide.computer/product/specifications
Then yeah, it's ridiculously expensive.
That said compared to competitors it's in the right ballpark, but I have no idea how companies manage to spend so much money for this stuff. I am the founder of my own tech startup and I remember when I was looking at storage solutions and building out computing clusters there were companies charging absolutely insane prices.
I literally just spent about a week of my own time studying and learning as much as I could about it on my own and ended up building out my own custom solution for about 20-25% of the price these other companies were charging. I remember hearing people trying to scare me out of it saying if I did my own solution I'd need to hire full time operations people, and I'd always have to worry about things breaking and maintenance or headaches and nightmares etc...
It's been over 10 years now and absolutely no headaches, no nightmares, and very very minimal maintenance is needed.
Some drills I practiced were setting up RAID 10 and on hardware failure having an email sent to me, so I went through the process of getting RAID 10 working using 4 of the drives, and then I would physically pull a hard drive out of the system as it was running to simulate a failure, and then I swapped in a new hard drive and ensured that the RAID rebuild process took place.
Once I was confident enough, I went to thinkmate.com and bought two JBOD expansion chassis each of which supported 78 drives and filled each of them up with 5 TB Seagate drives to get a total of 390 TB. The JBODs are managed by a 1U server running Ubuntu Server and software RAID using mdadm. I also bought 8 compute servers that were considered high density and could fit in 4U. I got a Cisco router for Internet and I networked everything within the data center together using a used Infiniband switch that I bought off of ebay. I also got a KVM so I could remotely access all of the systems. If there's one thing that I would change today, it would be to use ZFS.
I remember comparing the cost of doing this DIY setup to some other premade solutions like EMC and the cost was astronomical, like 3-4x what I ended up having to pay. I also even remember watching Linus Tech Tips and Level1Techs on Youtube and they both had good content about how they managed their storage that was fairly reasonable but still slightly on the pricey side, nevertheless I learned quite a bit from it.
At any rate, the bottom line is that it's not trivial to learn all this stuff by any means and I remember having some serious frustrations due to just how bad and demoralizing some error messages can be, but it's not thaaaat difficult either and in my situation my company is self funded, no venture capital or outside investors of any kind so every dollar my company spends is a dollar out of my own pocket. You better believe when I'm starting my business I'm not going to just blow 100s of thousands of dollars extra unless I absolutely have to.
About 5 years ago I managed to use my own storage system to ditch Dropbox in favor of Nextcloud, which is just leaps and bounds superior. I remember I got so frustrated with Dropbox because I wanted to just do something as simple as create a sub-directory and grant only read only permissions to some accounts. I also remember wanting to do some simple things like create a fake account with very limited permissions that could be used in our conference room for presentations or demos but the only way to do that with Dropbox would be to create an actual account and have to pay the full price for it.
Nextcloud works amazingly well, has all kinds of cool plugins, and gives our organization a lot of flexibility that Dropbox doesn't and I can make as many accounts as I feel like for whatever reason I feel like.
And not a single mention of backups.
Good luck with that.
I mentioned I use rsync for backups.
So different backups happen in varying directions, with the colocated servers using the main storage system to back themselves up, and the main storage system in turn backing portions of itself onto the colocated servers.
Not everything gets backed up, however, and most of what does get backed up can be compressed quite significantly. We mostly just back things up that can't be recovered from third parties or that are proprietary to our organization.
See pictures: https://pbs.twimg.com/media/FfT7MHoUoAE90QZ?format=jpg&name=...
That strikes me as being in the right ballpark, but it's going to be tough to swallow since that's the lowest level of granularity.
For most orgs you'd be left paying for a lot of excess capacity you couldn't immediately put to use as you migrate workloads in. I guess in ~4 years once you reach steady state and you're retiring / replacing these things it all works out, but if you're migrating from vmware or something else in a traditional blade/chassis world it's not like you can just wave a magic wand and move $500k worth of compute over to this thing at once.
If you're green fielding something, that's a lot of cash to sink in on compute you may not need for some time in the future. Never mind your DR site(s) also needing that much...
I'm kidding, mostly. The engineer in me is bothered by it, but the part of my brain that cares more about design and branding understands that 0x1de would cause inconsistencies in the branding elsewhere. (E.g. 0xDocs: https://docs.oxide.computer)
We did manage to get a neat PCI vendor ID: https://pcisig.com/membership/member-companies?combine=01de
from https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...
"One size fits all" will never work.
Here are the main headaches that someone who actually runs a data center will run into:
- How many power circuits does the DC provide you? What voltage? AC or DC?
- How many amps do the circuits have? Is the PDU provided by the DC, or do you provide it?
- How many upstreams will you have? Dual core routers? BGP? Static routing? Failover?
- Upstream provided via Fiber? Copper?
- Given that for the last couple of years being able to re-buy a typical server model (say: Supermicro), CPU, Storage for more than a couple of weeks has been a challenge, and given that rack hardware is supposed to run for years - how can I be sure to be able to scale?
- What happens if that Oxide startup goes bust 6 months from now?
- Would any sane DC operator really buy "we just built our own switch"? I wouldn't. Those who have been in that business for 20 years still f*ck it up on a regular basis. I want a proven vendor for my mission critical stuff.
I am sorry, I don't see them addressing ANY of the real life pain points you have operating your own rack. Unless they ship a couple of data center engineers within the box.
The whole cloud stuff is simply designed for managers who believe that in a job market where it's hard to find DC/server engineers that things will be magically done in the background. With this, it's worse - it's pretended that "stuff just works" if you ship it in a single box. And that's simply not a case. A DC installation not constantly maintained by skilled engineer will go down in flames within months, one way or another.
By the way, the list of questions would go on. At least over here, DCs offer different rack footprints. They don't even list their footprint.
But most importantly: The TFM doesn't say "There is an experienced network engineer in the box that you will own and that will do the maintenance". Unless this is provided, this will blow up.
Again: I can imagine that they find a hand full of managers buying into this. Up until the date the first few of their racks have burned down a DC, or got hacked and all customer data stolen.
Come on, this is hacker news. Everybody here should know that nobody so far has found a way (not even via AI) to replace the competences many of us here have. Again, I understand that management would love to get rid of those expensive and annoying nerds, but the ACTUAL REALITY is: An unmaintened rack in a DC will go down in flames, and management people will be in tears.
So, let's just be realistic: Oxide is simply a terrible idea that will blow up. But I congratulate them for VCs falling for this.
Now, let's find a VC that is funding someone doing an IPMI system that is not shit, does not have access to all data of the server, and does not get hacked within 5 minutes after deploying it. That would address an ACTUAL need.
Sometimes it makes sense to ask those who you hope your future customers could be.
Yep, exactly. That's precisely the reason why this "solution" to a problem that doesn't exist is currently being praised as the soon to be heir of the throne of computing
EDIT: "it's not for me" - I'm not working with organizations that have that sort of need directly. Re-reading that phrase, it came across as a bit dismissive of what they've built (which is undoubtedly impressive).
And yeah it's totally chill: this product is not for everyone. No slight taken!
Just my perception given the front page. I agree it's a bit vague.
None of that applies here.
Unless you rent it out and are a cloud provider, but the website at least does not seem to target those.
Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
Hybrid clouds even means devs might not know whether it sits in the corporate data centre or a public cloud, because it could be either/or depending on current capacity.
> Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
"You" as in "the organisation as a whole" don't get elasticity. "You" as in "your department" or "you as an individual" do get elasticity.
Right, but to the degree that you get elasticity, it starts to look more and more like "someone else's computer", no? If multiple people/departments/etc are provisioning virtual instances on one shared cloud infra, with nobody who's using the provisioning API caring about the underlying capacity (and capacity is planned indirectly by forecasting, etc), then it really starts to sound like "someone else's computer" to me. That "someone else" just happens to be another org within the same company.
I've spent the last decade building on-premises systems very like what Oxide is doing, but I've had to build them out of stacks of servers, switches, storage appliances and VMWare licenses. And the network cabling, and fan noise, and the number of power cables, and.. oh man, I can't wait to install one of these things myself. Having a single point of responsibility for the whole thing shouldn't be underestimated either - I've spent far too long trying to resolve problems with vendors on both sides blaming each other.
It's worth mentioning too that building something equivalent to this would be across more than one rack, and easily cost in excess of $1M.
That might be true historically, because the only way you could get resources provisioned on-demand via an API from someone else who'd built it. You had to run in someone else's datacenter to get the capability which you actually wanted.
Times have changed. Now, businesses think about "Cloud compute" as being synonymous with "on-demand", "elastic" etc. Where the actual silicon lives is merely an after-thought.
> Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
Buy enough of them and you will :)
If you have to buy the silicon and plan capacity for it (as in the case with Oxide for example), then it cannot be an afterthought. Which is exactly why I would not consider it cloud computing.
Someone's always got to buy the tin and manage that. Some people are big enough that they might get a benefit from doing that themselves, rather than having Jeff Bezos do it for them.
From the application team's perspective, call API then container go whirr.
Of course you do, right up until the point where you’ve used all available capacity. Just like with public clouds (ask anyone using meaningful amounts of {G,T}PUs). Elasticity doesn’t imply infinitely elastic, that would be ridiculous.
But that is literally not possible with hardware you purchase yourself.
Sure, you can buy X amount of hardware, and provision up to X amount of virtual hardware via an API, but then what? You can't provision any more until you go and buy more hardware. This is why "cloud, but local" is a contradiction IMO. You can only be "cloud-like" if you're under-provisioning. The moment you want to actually use all of the capacity you already paid for, you're not a cloud any more, because you've provisioned all of it.
No elasticity, no cloud.
Sure it is. I think TFA is talking about a company selling you exactly that capability.
> You can't provision any more until you go and buy more hardware.
But this is also true of AWS etc. When their estate gets full, they need to go buy more hardware. Regardless of who owns the tin, someone's doing a capacity plan and buying hardware to meet demand.
The point of 'cloud' is that you move that function out of the domain who are actually using resources to solve business problems, which is where it traditionally sat. Historically, if you wanted to run a service, you had to go buy some hardware and hire someone to manage it for you.
A cloud-like model means that the application engineers no longer care about servers, disks and switches. Instead, they just use some APIs to request some resources and then deploy a workload onto them. The details of what hardware, where and how is fuzzy and abstracted. Or cloud-like.
> You can only be "cloud-like" if you're under-provisioning
Everyone under-provisions. Nobody runs at 100% utilisation.
The definition of cloud is “out there in the sky - not here”
and that isn't it...
You may not like it, but it's been a long time since "cloud" has exclusively referred to "someone in another organisation entirely runs the computers" as opposed to virtualised allocation of resources.
I suspect that while I do appreciate how some posters upthread find the website a tad on the vague side, the target customers-in-potentia will understand it fine.
Define "I"?
I-as-developer can call an VMware/OpenStack API with an on-prem/private cloud and get a new instance just as easily as calling an AWS API. I-as-developer does not have to worry about elasticity if the IT hardware folks have the capacity.
From a developer's perspective an API call is an API call, and to them it's just another instance.
As an engineer, an Oxide system works like any other cloud provider. You’re just interacting with its API and tooling like you would with Google Cloud or AWS.
To someone on the IT/Operations side, obviously there are differences but theres SIGNIFICANTLY less labor required to build-out and operate an Oxide system vs a rack full of servers. The biggest difference for these people is that there’s actual hardware vs a Cloud Provider, but also costs are fixed so there’s likely no monthly or quarterly meetings with finance arguing over the cloud bill, tying people up to try and shave a few thousand off the bill every month.
In finance/accounting, Oxide is probably the most different: now compute is CapEx rather than OpEx. Depending on your company’s stage that can be a wonderful thing for the bean counters.
But it’s also gonna be much more restricted. So I guess one could see as kind of “Apple for data centers”? Have a nice appliance and be happy as long as it runs as it should (but hope it never stops working as it should).
What they're selling is building blocks for private/hybrid/public clusters that may or may not fit your definition of cloud depending on where they happen to be located but where what the term signifies is that it is built to be a building block of a cloud setup that includes features above and beyond a "regular" server to provide what most people tend to associate with a cloud. That is, you're getting APIs to spawn virtualized compute and storage, rather than having to install a hypervisor and management APIs etc. and combine the resources into a cohesive whole yourself.
My guess is that most of them will end up being sold to either hosting providers to provide cloud services to their customers or companies large enough to operate their own data centres where the IT department will use them to offer cloud services to other parts of the business, where the term really is not misleading anyone.
It's too big even for most "private clouds" in smaller companies, because they tend to be too small to be able to order by the rack even when there are APIs etc. offered to developers to provision compute and storage (e.g. years ago I used to operate a hybrid cloud setup with a ~1000 or so VM instances across several countries, and while we had several racks worth of gear we physically owned in aggregate, we didn't have any full racks in any individual location).
The Oxide machines seem to be aimed at 2020.
That being said I'd still expect a monthly service fee for networking, electricity and service in general.
> “Oxide is a strong believer in the need for open-source software at the lowest layers of the stack -- including silicon initialization and platform enablement. With the availability of AMD openSIL, AMD is showing that they share this vision. We believe that the ultimate beneficiaries of open-source silicon initialization -- as it has been for open-source revolutions elsewhere in the stack -- will be customers and end-users, and we applaud AMD for taking this important and inspiring step!”
* https://community.amd.com/t5/business/empowering-the-industr...
See Bryan Cantrill's (Oxide CTO) presentation on their adventures in this space:
* https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
* https://news.ycombinator.com/item?id=33145411 (discussion at the time)
They are likely looking at future racks with GPU as well, but as a first product getting the basics right makes more sense.
Nvidia GPU drivers are very proprietary, which means that admins and developers have limited visibility into them if they misbehave in any way. This goes against Oxide's philosophy of full visibility into a system that you purchase.
Nvidia's CUDA software has a significant lead ahead of AMD and Intel GPUs, and they're not going to open source it any time soon. But this is a rapidly changing landscape, and AMD and Intel and others are pouring an enormous amount of research into getting their hardware and software to match what Nvidia has going. Nvidia is in pole position, but they're not guaranteed to stay there.
There's still a large market for the CPU workloads that Oxide is offering. For now, Oxide will be concentrating on meeting this traditional compute demand. But you're right to point out that in 2023, the absence of a top tier GPU in these racks is noticeable. I suspect Oxide will want to include some form of GPU or TPU into the next version of their system, but they won't just grab whatever hardware happens to be in fashion. It needs to work with their system as a whole.
This is where they will enter the real “hard” part of hardware, with an exec team from software. Can they respond to the market while making hardware?
They seem to have presented as generally thoughtful about their approach. If they can release major variants about annually or even sub-annually that is what I think will enable them to win.
If you take some time and read around their site you'll see that they're offering a ready to run (turnkey) server. They have everything packed together, they've integrated everything that is needed (CPU, disk, networking...) into a nice looking cabinet, with not too many wires and they're selling it as a complete package.
If you're in need of a server (cloud computer) and you don't want to but separate components and unpack them and connect them yourself, then this looks like a possible solution.
Oxide Cloud Computer
No Cables. No Assembly. Just Cloud.
Contact Sales
How much easier can they make it? They clearly want to sell computers.
I mean, the former is just the latter with some of the setup done for you no? Anyways, it’s a full server rack, with tightly vertically integrated hardware and software. Not sure if you’ve poked around the rest of their site, but it seems like their whole software stack is designed with some really nice usability and integration in mind: there’s a little half-snippet there suggesting that provisioning bare-metal VM’s out of the underlying hardware could as trivial as provisioning an EC2 with Terraform, and if that’s the case, that’s _massive_.
> And if it's a server rack, how come you don't need any cables?
Because they’ve gone to great lengths and care to design it to not need anything extraneous IIUC. I think the compute sleds all automatically mount into some automatic backplane that presumably gives you power, cooling and networking, and then, as above, you presumably configure all that via software, as you would your AWS setup. Not an expert here though, happy to be corrected by anyone who actually knows better.
> Do they host it in their data center or deliver it to you?
Presumably the latter, given they’re a hardware company, but if their software is even a 10th as good as it seems, I fully believe there’ll be a massive market for renting bare-metal capacity from them.
You use words like: it seems, I think, presumably, I believe... This is what we're arguing. A company that has raised $44 million Series A for sure can afford to clearly write what they offer.
I understand, you can't have all the people happy and no matter what you do there will always be "weirdos" that don't like your page/design/wording, but hey at least recognize it :-)
We have a terraform provider, yes https://github.com/oxidecomputer/terraform-provider-oxide
In a sense. Yes, it is a rack of servers. You're buying a computer. But we've designed the rack of servers as a full rack of servers, rather than an individual 1U. Comes with software to manage the rack like you would a cloud; you don't think of an oxide rack as individual compute sleds, you think of it as a pool of capacity.
> And if it's a server rack, how come you don't need any cables?
Because you are buying an entire rack. The sleds are blind mated. You plug in power, you plug in networking, you're good to go. You're not cabling up a bunch of individual servers when you're installing.
> Do they host it in their data center or deliver it to you?
Customers get them delivered to their data center.
Happy to answer any other questions.
It wasn't obvious to me you can own the stuff where it runs (I hope I understood it correctly)
So it is big, it looks good, but no specs. What architecture ? Some "cinebench" numbers ...
Dimensions H x W x D
2354mm (92.7") x 600mm (23.7”) x 1060mm (41.8")
Weight
Up to ~2,518 lbs (~1,145 kg)
BTW, they are following some no pictures policy. I found only a few pictures of boards but no picture of the product as a whole.
The key difference is the price structure -- IBM leases the hardware wherever they can get away with it, and uses a license manager to control how many of the machine's resources you can access (based on how much you can bear to pay). This, however, is like a mainframe you own.
* Physical space - The servers themselves require a certain amount of room and depending on the workloads assigned will need different dimensions. These are specified in rack "units" (U) as the height dimension. The width is fixed and depths can vary but are within a standard limit. A rack might have something like 44U of total vertical space and each server can take anywhere from 1-4U generally. Some equipment may even go up to 6U or 8U (or more).
* Power - All rack equipment will require power so there are generally looms or wiring schemes to run all cabling and outlets for all powered devices in the rack. For the most part this can be run on or in the post rails and remains hidden other than the outlet receptacles and mounted power strips. This might also include added battery and power conditioning systems which will eat into your total vertical U budget. Total rack power consumption is a vital figure.
* Cooling - Most rack equipment will require some minimum amount of airflow or temperature range to operate properly. Servers have fans but there will also be a need for airflow within the rack itself and you might have to solve unexpected issues such as temperature gradients from the floor to the ceiling of the rack. Net heat output from workloads is a vital figure.
* Networking - Since most rack equipment will be networked there are standard ways of cabling and patching in networks built into many racks. This will include things such as bays for switches, some of which may eat into the vertical U budget. These devices typically aggregate all rack traffic into a single higher-throughput network backplane that interconnects multiple racks into the broader network topology.
* Storage - Depending on the workloads involved storage may be a major consideration and can require significant space (vertical Us), power, and cooling. You will also need to take into account the bus interconnects between storage devices and servers. This may also be delegated out into a SAN topology similar to a network where you have dedicated switches to connect to external storage networks.
This is what Oxide has to offer. They have built essentially an entire rack that solves the physical space, power, cooling, networking, and storage issues by simply giving you a turn-key box you plant in your data center and hook power and interconnects to. In addition it is a fully integrated solution so they can capture a lot of efficiencies that would be hard or impossible in traditional design.
As someone with a lot of data center experience I am very excited to see this. It is built by people with the correct attitude toward compute, imo.
The main USP: you own it, you're not renting it. You're not beholden to big cloud's pricing strategies and gotchas.
Secondary USP, why you buy this rather than DIY / rack computer vendor: it's a vertically integrated experience, it's preassembled, it's plug and play, compatibility is sorted out, there's no weird closed third party dependencies. Basically, the Apple experience rather than the PC experience.
It's the iPhone of hyperconverged infrastructure.
(Sorry; three sentences.)
The industry moved to commodity x86 for compute for a reason.
[0]: https://github.com/cbiffle [1]: https://cliffle.com/p/dangerust/
Can we please stop with the "the only way to get our pricing is by booking a sales call with us." This is a 100% surefire way for me to never pay for your product, and instead go to competitors who provide straightforward, no-nonsense pricing on the website.
This is ironic given the amount of self-praise they give themselves in this article about how much they care about shipping something you can buy once instead of renting from the traditional cloud. Great, so then tell me how much it costs...
I worked photojournalism in university, and it really strikes me when people talk about how cool their novel interconnect is and how clean the design is, but their long writeup doesn't feature any pictures?? Would love to see what it looks like.
(emphasis mine)
I'm having a hard time getting past the first sentence of this blog post.
Maybe I'm missing the obvious.
And these offerings have existed for years.
And from hardware manufacturers, Dell/VCE, Nutanix, and more "hyper-converged" infra has existed.
Note: I'm not being a hater. I'm just genuinely confused.
---
https://www.oracle.com/cloud/cloud-at-customer/
Wouldn't you still be using 3rd party vendor software, it'd be Oxide software now?
AFAIK they manage it for you, as if you're just a colo. Whereas oxide just hands the entire rack + software over, but with no self install of any software stacks required (such as with azure)
That's literally why it's so exciting lol
I feel like the "running your own datacenter" complexity of on prem is a big part of why companies go to the cloud in the first place - but that a good amount of cloud lock-in is actually in the software/service stack and reliance on it. I know there is a competent open source option for most of these services (and in fact most are repackaged open source software...), but they do seem pretty disparate and require quite a fair bit of knowledge and expertise to deploy and run at scale.
Secondary question in the same thread: What about "accounting"? Running infrastructure at large-scale companies (such that would look at a rack-level solution) usually entails budgeting and accounting of resources - how does oxide simplify this for infrastructure ops trying to keep the various IT consumers honest?
On accounting: yes! This is something that we have designed abstractions for from the start with respect to tenancy with our silo and project abstraction[0], and we are actively adding resource allocation and monitoring; stay tuned!
[0] https://docs.oxide.computer/guides/key-entities-and-concepts
It's high time the datacenter takes a more holistic approach. For years I have been thinking that it's silly to stick a bunch of tiny fans in individual servers when one large impeller pulling air from an inclosed rack would be more efficient. Imagine large fans with ducts that come down to each rack or row of racks. Current in == current out; so a large diameter fan (5 feet) coupling down to small hose (10 inch) means the air moving through the small end will be really moving fast through those components!
Old school CIO's, who make enterprise decisions without developer support, are typically the buyer of hyper-converged infrastructure (or Utility companies who struggle with CapEx vs OpEd).
I wish them luck. There's a bunch really good folks over there.
Rust, Oxide, and iron-y.
Does this "lock" you into the Oxide platform and you just buy into the whole thing instead of buying some server from Dell, then running some Proxmox like software and a Docker host?
My dad, who worked for IBM through the 90s and 00s always points out how amusing this is. We started with the cloud. Then went PCs. Then are going cloud again.
Doesn’t mean this is wrong. It’s just amusing.
If we do it properly, it should be far more optimal than all the local computers not doing anything most of the day.
It's a company run by people who care and have cared for a long time, and they have all my faith.
Over the decades the separation between "software" people and "hardware" people has grown. With "cloud" people have grown comfortable papering over poor basic performance by abstracting away even your visibility into how a system is running. "You don't need to worry about that! It's ~serverless~." But you'll worry when that bill comes due at the end of the month.
With systems like Oxide, you are allowing users to once again actually see what they get, and ensure they get what they paid for, from a high-end cloud server.
It should be putting other systems providers on-notice that the days of flaky, non-performant, and poorly-integrated components are behind us. People want beasts from their servers.
And software designers, this is also your wake-up call that you can't just put lousy-performing software with poor CPU utilization and memory hogging on big metal and hope that it's "good enough." You really need to design your software to run ~efficiently~ in such systems.
Lol. Bro.
This is a hot take, but I think Moore's law/Wright's Law has actually been disastrous for the entire field of software engineering even while it's been an amazing boon to software businesses.
The guy’s “lol” was at the (possibly unintentional) joke.
This 100%!
Not everything needs to be a subscription. Sure there are running costs for operating that cloud computer as well as ongoing software development costs. But having photo albums in the cloud shouldn't cost me $30/month just because of storage - let me buy that hard drive (and maybe compute why not) and pay $10/y for operational costs
How many engineers and how many years does it take to build the infrastructure, write all the software, and deal with all the hardware to provide a photo service that "just works"? How many customers do they need before they break even? And obviously they can't raise VC because a business model that is predicated on making basically no money per customer can never have an exit. How would you bootstrap such a thing?
It feels strange that physical storage costs almost nothing on amazon but the storage attached to a cloud machine is expensive. But run the numbers and you'll see that running a cloud service isn't about hardware costs at all.
It seems that Oxide's product is rather tied to AMD's CPU offerings for the time being (which perhaps will serve them very well for now), and given what they've accomplished so far, I imagine it would take quite a bit of effort on the platform initialization side (and other lower level stuff) to get things working with Intel CPUs instead. Surely the ability to have vendor diversity for many of their components should be an advantage for Oxide (and their customers downstream as well), so maybe there's something to think about there. Of course, this is all interesting only once Intel actually gets competitive again on the datacenter CPU side of things, where they seem to have really dropped the ball in recent years.
On the other hand, their networking switch hardware uses Intel components (Tofino), so maybe that'll be the extent of Intel's integration in their rack for the foreseeable future?
AMD recently purchased Pensando, I'm not sure if their network chips are similar to Tofino but they seem like they are P4 compatible DPUs so they might be a good choice when migrating off of the dead Tofino platform.
So if anything Oxide will be moving further away from Intel!
I would love to work on this kind of stuff, not sure how to get started to end up working somewhere like Oxide..
It seems the only reason why people are getting too excited is because of the team members and the name.
What if this venture becomes yet another VC pump and acquire scam to get you to purchase a massive room heater?
They better not get themselves acquired like the rest of the hyped up VC scams out there.
Best case this gets acquired by HP,IBM or Dell and will die out because talent will leave.
I know a 4bn/y revenue company that could greatly benefit from this but they will never even consider buying from a company like this.
From their press release:
> Oxide customers include the Idaho National Laboratory as well as a global financial services organization. Additional installments at Fortune 1000 enterprises will be completed in the coming months.
* https://www.bloomberg.com/press-releases/2023-10-26/oxide-un...
From an interview:
> And what was the reaction of lucky rack customer #1? “I think it’s fair to say that the customer has appreciated the transformationally different (and exponentially faster!) process of going from new rack to provisioned VMs.”
> The same day Cantrill appeared as a guest on the Software Engineering Daily podcast.[1] (“The crate, by the way? Its own engineering marvel! Because to ship a rack with the sleds, it’s been a huge amount of work from a huge number of folks…”) Cantrill wouldn’t identify the customer but said that “Fortunately when you solve a hard problem like this and you really broadcast that you intend to solve it… Customers present themselves and say, ‘Hey, we’ve been looking for — thank God someone is finally solving this problem’.”
* https://thenewstack.io/in-pursuit-of-a-superior-server-oxide...
Niches can be profitable. Not every company has to be "web scale".
> I know a 4bn/y revenue company that could greatly benefit from this but they will never even consider buying from a company like this.
What does "like this" mean? Small? Every company starts out that way.
I think its cool that someone is trying to do startup hardware things, as in recent years it seems like its mostly software. I don't a use for the product in my area of IT, but I wish them luck.
https://www.asrockind.com/en-gb/4X4-7840U-1U
Also a former employer who was a bit below the hyperscaler level also had what they called "roll-in racks", though I believe they were just taking standard servers and networking gear and integrating them somewhere (presumably with cheap labor) and then bringing them into the DC as needed.
So I don't think it's a particularly new concept, but I agree it looks like it has some potential as a product.
Also, I remember for example HP BladeCenter BL20p systems. There you would have up to 8 or 16 servers per 6 rack units IIRC. So about 56 to 112 servers in 42U.
If you already get a 50% discount on your on-demand prices for AWS, does buying your own cloud make economical sense?
You're not buying a bag of peanuts and they will likely price it differently if you're buying 1x or 500x.
What is so different about CapEx and a bag of peanuts that defies even providing an approximate range? Number of zeroes, no?
I don’t like it, but it is the norm sadly. It’s just a signal for “it’s expensive and if you even care about the number we don’t want to talk with you”.
It is not, but when it's written publicly you cannot send different prices to different customers, depending on who the customer is
I just hate calling people to determine if we’re going to match, when they could have given me that information up front.
All that really matters is whether Oxide's guesses about who'll buy and what they'll pay are right.
Lets say that Oxide are guessing that their lead customers are telco and oil/gas: conservative industries, regulated, very strict about uptime and data sovereignty, etc. Oxide might spend 1 to 2 years running the gamut of all the people in those types of customers that can say no: ops, swcurity, network teams etc. But if Oxide can reach the finance team and get them to believe that a one time chunk of $$ is better than opex in the current interest rate environment, them it really doesnt matter what the tech stack is. (And bear in mind that financial models are f(t) so things might turn upside down as the economic picture changes over time)
Assuming they have done this math, (surely a yes, or else VCs would not be lending them money) Oxide must need sufficent funding for a long sales cycle, and plausible enough bench strength to support customers who are going to hold them to ruthless SLAs and penalty clauses. All of that take serious coin up front, eg theres no outsourcing your 24x7 support to some kid reading from a script.
The Cloud is not "a computer". The Cloud is a public utility. You don't rent the transformers at an electric utility company - because why the hell would you? It's just one component of a much larger system that is valueless to the consumer without all the other components.
What these people are selling is more like a battery that you can purchase that plugs into the electric grid. But why would you want to purchase a battery? It degrades over time, losing value, and eventually needs to be recycled, and somebody has to deal with all that. Renting is the perfect way to push all of that time-consuming and complex maintenance out to someone else who can lower cost with volume.
The idea that renting servers is not sustainable would suggest that somehow computers are not like housing, cars, shop vacs, or any of the million other things you can rent. A computer is an expensive commodity like any other, and renting makes perfect sense a lot of the time.
If you need to purchase, for economic, load, tax, regulatory, or other reasons, there are already ready-made computers with (or without) service contracts and supported OSes that can be plopped into any colo, and colos sell dedicated machines already. This pitch doesn't include anything new other than buzzwords. Yeah I'm sure they did a bunch of engineering - that nobody will ever need. I'm calling shenanigans.
Didn't that still come with the weeks of overhead that compelled people to adopt cloud in the first place?
I'm not sure what "weeks of overhead" you're referring to, specifically, as it could mean a few different things. Happy to elaborate if you will :)
Dude, why are you even in this thread? Trying to be a Debbie Downer is just going to ruin the rest of the day.
Trying to get the last word in on the Internet is a quick and easy way to have a bad time. Let it go and move onto something else.
My google-fu failed to find any articles for or against my statement that weren’t paid advertising or lightweight tech summaries. StackOverflow will have to do. https://stackoverflow.blog/2023/02/20/are-companies-shifting...
If the opex model doesn't work for an organization, that they perhaps just went above that utilization threshold where it doesn't make sense anymore, and would be a potential 0xide customer?
When opex is a primary benefit of the cloud, that's specifically for start-ups and other businesses that have little working capital. The actual primary benefit of the cloud for most cloud customers is flexibility - prior to AWS adoption, getting a server provisioned could be a multi-week if not multi-month affair negotiating between devs or operators and on-prem infrastructure workers to get the servers through capacity planning and provisioning. AWS made provisioning so simple, that you could start to set up auto-scaling, because you had extraordinarily high confidence that the capacity would be immediately available when your autoscaler tried to scale up.
But opex is not a benefit of the cloud for heavy/established businesses. Cloud opex is a financial expenditure every month that you can't get rid of and counts directly against your profits. Indeed, the desire for capex in the cloud is so high that businesses routinely purchase Reserved Instances and other forms of committed usage, which allows accountants to treat the cost of the RI as capex and then discount the expenditure through depreciation (to zero, since there will be nothing to sell when the RI expires) over the lifetime of the RI. It is normal and frequent for businesses to make capital expenditures to reduce their operational expenditures over time, thus increasing their monthly/quarterly profits.
Oxide's unique value proposition is to give customers, particularly those with high monthly cloud bills that they have difficulty reducing, the operational flexibility of cloud computing with the profit-improving benefits of capital expenditures.
Surely the main benefit of reserved instances is lower TCO and if you can show you can afford AWS for 3 years a bank would surely loan you the money to pay upfront if you can save say 50% it is simply cheaper even with interest.
For a regional hospital for example, there might be a desire for in-house network and hardware - but perhaps the system for digital patient journals run on Kubernetes and managed databases.
I'd rather have $1 in capex than $10 in opex
1. Are there integrator companies that take these building blocks and build datacenters for other companies?
2. Are these racks intended to be bought in large quantities and put into new datacenters?
3. Is liquid cooling an option?
4. Do you sell datacenters as well as just the racks? Do you have any of your own datacenters built out?
5. Tough question: if these are so competitive and good for your customers, why aren't you also making lots of them for yourself and also selling cloud compute?
2. Eventually, right now we are just starting, so working closely with customers and in small quantities, but eventually that is the idea.
3. No. We are already doing very well with fans, the design means a very tiny power draw. The racks are quite quiet.
4. No.
5. Selling hardware and running a private cloud are two different businesses. The public clouds don’t also sell servers. We don’t plan on running a cloud, even if doing so would be a nice experience, we have to focus on what we’re doing.
1. Seems like you would need quite a bit of capital to be one of these integrator companies, to float the cash to buy land, build a facility and buy the racks. Does your own vs rent philosophy extend so far to also say that companies should also own the land and facility on and in which a server rack sits? Early on, Amazon talked about purpose-built datacenters being important, with huge power and internet connectivity requirements, earthquake/fire/flood protection and HVAC requirements. Sounds like the best way to go, if you have a huge amount of cash, seems like unit economics are now favoring data center costs in the billions of USD.
2. For now, would it make much sense to buy one or more of these and co-locate it with a cloud company's existing facilities in an office building? Or, if you had some space in a "good" datacenter where you leased space and sublet space, connection, power and maintenance costs to your customers, would that reduce the friction for buying one of these? Or does that essentially defeat the purpose and advantage of owning your own racks? If you remove the price of renting the rack, what's the ongoing cost to park one of these in an existing datacenter? Is that even an option?
3. From a purely mechanical perspective, the performance and efficiency advantages with liquid cooling vs. air cooling seem great. Seems like the problem is mostly around installation, maintenance, application requirements and reliability. Would be great to eventually see an integrated rack with liquid cooling plumbing hookups going directly to heat exchangers. At the cost of massively increased initial cost, maintenance cost, and with a built-in severe, catastrophic failure mode... Understood that this doesn't really make sense if these racks are being put into existing air-cooled datacenters or office buildings, which seems like it's going to be the case for awhile?
5. Too bad, seems like your prospective customers are in a similar sales funnel as people who want to rent capacity in a private cloud (and it's always nice to be your own customer!) What prevents your competitors from also selling servers? Do you think that the reason that public cloud companies don't sell servers is that they think their competitors will get ahold of the hardware and copy their technical advantages? Or is it that they have the deep pockets for building datacenters that their customers don't necessarily have, and they would rather have monthly revenue anyways?
3. Software engineer, not hardware, haha. I don't know if people have tried liquid cooled racks, personally.
5. "What prevents your competitors from also selling servers?" I mean, like I said, they're just totally different businesses. Doesn’t mean it’s impossible, but it feels like there’s not a ton you’d be sharing there, other than one half knowing where they’d be getting some supply from.
Virtually all the value we get from cloud is in their managed offerings of complex and difficult to admin systems like Kubernetes, Postgres, and managed storage layers.
That’s where all the value is but it’s also where all the cost is. If you just want compute, storage, and bandwidth all those can be had at commodity prices. Look at Hetzner, Hivelocity, FDCServers, DataPacket, OVH, or VPS providers like Vultr. You can get dozens of cores, terabytes of SSD, and gigabits of unmetered bandwidth for a few thousand dollars or less. It’s very cheap.
Without the high level managed stuff we would be off cloud five minutes from now.
I’m sure there is a market for this among people who run on premise data centers, but I think they would have a much larger market if they went further up the software stack. Right now they just look like they are competing with Dell and Supermicro, not Amazon or Google Cloud.
https://azure.microsoft.com/mediahandler/files/resourcefiles...
The difficulty is that HP/Dell/NetApp/etc are well established and maybe if you're a SME you can appreciate the benefits of an end to end integrated solution but most of the people making purchasing decisions largely don't. It's easy to market on CapEx but difficult to market on OpEx - you're asking customers to (likely) pay more for a similar performance invariably less featureful solution from a small company which they're then locked into largely leaning on the promise it'll be easier to operate (where is your GPU integration, object storage, shared filesystem, integrated kubernetes, etc). They then have to convince all the incumbent software vendors to support their software (e.g SAP/oracle/etc) on this specialized hardware.
Do you think people want to design games with unreal engine on the cloud? Is there no desktop application that's safe? I don't fully accept this premise and I wonder if I'm the crazy one sometimes.
> TODay WE aRE aNNOUNCING tHe
It's actually much weirder than that, as some of the capitals in my quote are actually big lowercase letters, not capitals (i.e. the opposite of smallcaps).
Open it in a private window and see if you can reproduce. By default, a private window uses no extensions though I prefer to have uBO enabled by default there, too.
Another peculiar detail I saw on landing page is screenshot of management interface. It must be old as it shows various VMs including FreeBSD 12.1 and Ubuntu 20.10. Ubuntu 20.10 is EOLed long ago and also wasn't a LTS release. Not something you wanted running in enterprise 3 years ago, nevermind recently.
My favorite is the automated CIO repo: https://github.com/oxidecomputer/cio
I'd love to see a video exploring the rack and showing the first time setup.
> While the software is an essential part of the Oxide cloud computer, what we sell is in fact the computer. As a champion of open source, this allows Oxide a particularly straightforward open source strategy: our software is all open.
But their homepage says this:
> As soon as power is applied to an Oxide rack, our purpose-built hardware root of trust – present on every Oxide server and switch – cryptographically validates that its own firmware is genuine and unmodified.
That sounds like tivoization to me. The value of their software being open source is heavily diminished by their hardware refusing to run anything but their "genuine and unmodified" software.
Can it run Linux or Windows Data Center as ex-Sun folk they seem to have loved the last open source release before Oracle but now is still community developed variant of SunOS ???.
I would imagine the system architecture is different enough that running Linux as the base OS would take some work, for example it doesn't have an AMIBIOS.
Would be interested to hear your experience.
So far, I’ve seen mixed reviews. Some say that you can end up using unsafe a lot and so it’s better to stick with C or even use Zig.
Wondered if there was any merit to this.
Using unsafe in Rust isn't inherently bad. If you're doing embedded work, using unsafe is mandatory once you get to the level of interacting with the hardware.
That doesn't mean the code is literally unsafe, just that the interactions are happening in a way that the compiler can't guarantee. That's completely expected when you're poking at hardware registers.
You still get most of the benefits of Rust, the language. I've had good success with it.
And so I was wondering whether using it is still worth it.
Obviously, it probably still has many benefits over C but something like Zig might be simpler and better suited.
No, you still get the benefits where it matters.
The "unsafe" is basically marking a boundary where you're doing things outside of what the compiler can verify. If you're poking at hardware registers, that's expected and normal.
Putting "unsafe" in a program at the hardware boundary doesn't reduce the benefits of Rust elsewhere in the program.
Specs of the switch are available here: https://oxide.computer/product/specifications
There doesn't seem to be a way to provision a bare metal operating system, so HPC is out, and the networking is previous-generation, so there are two opportunities for progress right there.
Now that they're VC-funded I expect an OEM to snap them up before either opportunity can be pursued.
I remember a time when a Series A was rarely above $10m. Feels like at that capital level they should be further into the alphabet.
[PDF] https://web.archive.org/web/20050325213243/http://www.egener...
The same hardware and software using kvm would be much more appealing. Not being able to run nested virt or GPUs on such a powerful and expensive rack is pretty brutal. Perhaps that is planned for the future though.
It's going to be tough sledding getting these into enterprise accounts.
Best of luck.
This feels like the rackspace approach
But yes, if multiple teams and they have direct access this rack to provision VMs, maybe yes.
Also - as a developer you don't provision VMs on daily basis. You start a project, you need those resources at the very outset and that's about it?
Or maybe it'll be a curious unintended match instead, with whatever "streaming" devices Microsoft pushes next being connected to one's own "cloud computer" instead...
I can imagine SaaS businesses offering their product as a "box": making on-prem significantly easier. Deliver your software along with the hardware, hook it into their network.
Done.
Anyone with more experience have thoughts on if this will potentially become a common option for SaaS?
Similar hardware for home labs and for personal computers with open firmware would be great. I'd also love to see servers with hardware which can be upgraded (rather than completely recycled).
If you want kubernetes, you'd run it yourself on top of what we offer, think EC2, conceptually.
Here's our API docs if you'd like to check them out: https://docs.oxide.computer/api/disk_list
How is this different than colo space that has been predominant for years? Perhaps simply that it can be purchased more easily and standardized like a typical cloud VM?
I imagine colo’s have something like this, but I _suspect_ operationally it’s a lot more powerful, easy to use and functionally closer to the API’s and behaviour users are used to on current cloud providers.
Now, if someone who actually knows for sure feels like chiming in, I’m happy to be corrected.
Yes, companies are really going to trust a startup for a stack like this. Will they go all in for 100% of their infra? Of course not, but as a test against a similar infra stack at a lower cost with an appealing feature set? Why not?
They optimized for power. That roar of fan noise doesn't necessarily mean they're doing a better job cooling.
Wow. It makes sense if I think about it; heat is the primary byproduct and therefore a typical server rack is chock-full of chunks of spinning copper coils and plastic fins. But still... wow.
The Oxide rack is really the opposite in all of these dimensions: we do our AC/DC conversion in a power shelf and run DC on a busbar; the geometry of the sleds allows for very efficient 80mm fans; and (importantly!) every regulator is observable through PMBus and plumbed through our service processor and management network -- so you can see where it all goes.
And if you think I'm worked up about this problem, you should talk to sufferers of at-scale rack-and-stack who have been burned by this. ;)
> These supplies are often entirely dark
Good grief. What happens if a fan dies? The whole rack mysteriously shuts down? Or does the power supply just turn into a Minecraft lava block?
Did you consider other or additional options such as using rest warmth of DC to provide heating for neighboring city, or Frore AirJet?
Reinvent OpenStack, slap it on some 8Us and storage arrays, shove it in a rack, and ship it to a colo with a professional installer. So basically one of the larger server vendors but with integrated "cloud" software, minus the 2-hour service turnaround and spare parts.
The fact that they're writing the software from scratch is going to add years of lead time until they reach parity with other solutions. My guess is they're hoping they can get sticker price or TCO low enough that it outweighs the lack of functionality and uncertainty of a brand new everything. If you just need some VMs in a lab in the office closet, might work.
They only have $44M to build that company? I hope the next funding round is better :/
This does not accurately represent the amount of hardware design we have done, let alone the software work.
> The fact that they're writing the software from scratch is going to add years of lead time
What lead time are you referring to?
You all have arrived. Wishing you decades of prosperity and good fortune.
If users wanted to save some money, why not just implement a hybrid cloud system?
If you’re trying to implement a hybrid cloud solution, with part of it being on-prem, you would be in the market for hardware that to use in that implementation. We are now vendors of such.
The difference is you own the hardware.
It's not like commodity x86 gear with black-box (often buggy) firmware and layer-upon-layer of hacks and compatibility kludges to present the hardware interface of a late model IBM PC AT.
I can buy these at my local electroncis retailer. So pretty much commodity x86.
This is the type of "PC architecture" cruft many customers have been yearning to ditch for years.
Personally, I have always been annoyed that the BIOS is clunky and every change requires a reboot, taking several minutes. As computers got faster over the years, this has gotten worse, not better. At the core of cloud economics is elasticity: don't pay for a service that you don't use. Wouldn't it be great to power down an idle server, knowing that it can be switched on seconds before you actually need it?
considering you would still need to boot the VMs then, once the Oxide system is up, I’m not sure if this is such a big win.
And at a certain scale you’d probably have something like multiple systems and VMware vMotion or alternatives anyway. So if the ESXi host (for example) takes a while to boot, I wouldn’t care too much.
And, economics of elasticity - you’d still have to buy the Oxide server, even if it’s idle.
To be honoust, I'm using containers most of the time these days but even the full blown windows VMs I'm orchestrating boot in less than 20s, assuming the hypervisor is operational. I think that's about on par with public cloud, no?
> [...] vMotion [...] ESXi.
Is VMware still a thing? Started with virsh, kvm/qemu a decade ago and never looked back.
> And, economics of elasticity - you’d still have to buy the Oxide server, even if it’s idle.
That's a big part of the equation indeed. This is where hyperscalers have an advantage that Oxide at some point in the future might enjoy as well. Interesting to see how much of that they will be willing to share with their customers...
How do you mean Oxide might have that advantage as well in the future? As I understand, you have to buy hardware from them?
As to your question, I meant to say that as volumes and scales economies increase they can source materials far cheaper than regular shops. Possibly similar to AWS, gcs, Azure, akamai etc. It would be nice if they were able and willing to translate some of those scales economies into prices commensurate with comparable public cloud instances.
If you want more insight into all of the things that normally run on "PC architecture" - the 2.5 other kernels/operating systems running underneath the one you think you're running - https://www.youtube.com/watch?v=mUTx61t443A
I imagine they are aware that this isn't a solution for many customers. A John Deere tractor makes a poor minivan. This isn't for you. That's fine. It's not for me either. That's ok. I don't need to poo-poo their efforts and sit and moan about how it's not for me.
The BMC is often a huge problem, its a shifty architecture and extremely unsafe. Meta is paying for u-bmc development to have something a bit better.
Doing things like rack attestation of a whole racks worth of firmware if stacked PC is incredibly hard and so many companies simply don't do it. And doing it with the switch as well is even harder.
Sometimes the firmware runs during operations and takes over your computer causing strange bugs, see SMM. If there is a bug anywhere in that stack, there are 10 layers of closed source vendors that don't care about you.
Costumers don't care if its PC or not, but they do care if the machine is stable, the firmware is bug free and the computer is safe and not infected by viruses. Not being a PC enables that.
Oxide has put immense effort into writing open-source platform initialization code, and built their own open-source BMC/RoT solution.
Not in theory maybe, but in practice. Because as a customer, I would probably also need to put in immense effort to understand and maintain that software myself.
Until you've dealt with a malfunctioning Dell or HP server and have to live with being told "we don't know why it acts that way, we'll try to get the ODM to repro" I don't think you can appreciate how cool Oxide's offering seems.
Which would probably be a lot faster than trying to find someone who could maintain some non-standard firmware (as good as it might be).
Even if I had to replace the server on my own cost it would probably still be cheaper. And it would be easy to replace because it's commodity hardware, that was kind of my point.
Maybe if I had hundreds of servers Dell would have helped me out. At the scale of tens they told me to take what I got. The Customer got a lower performance solution and nobody anywhere could help them for any amount of money, short of replacing the gear.
That's just a performance issue. I've heard horror stories about reliability-- All the way down to disk firmware and RAID controllers. I consider myself lucky.
But how much effort (or money) do you think it would have taken to fix this issue if the NIC firmware was open source?
And with standard hardware, depending on the model, you might have had the option to add dedicated PCIe NICs for example. Not great, but at least something. Now try that with something proprietary (as in non-standard) like this Oxide system.
If they replace the broken component with a working component, then I don’t care how they fix their firmware errors.
All text seems to indicate the former while the logo seems to spell the latter?
If they manage to do get an IBM mainframe level after-sales support, they will rock.
Does someone know how much the lowest level box costs?
I absolutely dread that future. I want all my computation back to be local, doesn't matter to me if I own the cloud computer or not as long as I also don't own the environment it is embedded in. If the connection to it isn't mine and neither is its source of power then I don't really control "my" cloud computer.
For those companies this would probably be a cost effective solution since now we spend a lot of resources managing varying Openstack installations with different hardware from HP / Dell etc...
Pricing??
Also "Facilities" link in footer: 404
Would need CUDA for anything serious. Bound to Nvidia, for now. Even AMD and Intel cannot compete in this space.
"A disk cannot be added or attached unless the instance is creating or stopped." "A network interface cannot be created or edited unless the instance is stopped."
really?
Because IAM = Identity & Access Management
Want to spin up not only 1 but 20 instances via the console? Nope. Any kind of guest agent in the virtual machine? For direct interop? How is the guest filesystem freezed during snapshot creation? Checkpoints for incremental vm backup? They seem to use raw images, not qcow2? Why?
etc..
Not being negative ... just calling out this is offering is super late to the game.
Disruptors never have feature parity to established players in their first offering.
I know it's a brilliant team, and since ZFS/OpenSolaris/Dtrace I've known about Bryan. The product is a nicely looking product too. One thing I don't get is the target market. Who is it? Banks? Cloud vendors? Insurance companies? I think there's a part of the computer business that I don't get yet.
One guy in one of the previous HN threads here said that it's convenient to run 1 RFP for 1 box costing $1M-$2M vs running 30 RFPs for isolated components to put on the rack. Ok, I can see that. That's some argument.
But I know that Supermicro has rack assembly service. https://www.supermicro.com/en/products/rack and I'm sure Dells and HPEs have the same. Does it mean that it's impossible to call them and tell them: "Excuse me, I want 2 racks, 15 systems each, top of the line AMD CPU with RAM, NVMe storage, all maxed out, and whatever fastest Juniper switches you can find. Put me VMWare vFusion on it. Ship this thing to Infomart in Texas. I have $2M here with your name on it". They won't do this for me?
Its either this, or Oxide is going to e.g.: Bank of America, and along with RFP the bank is asking: "Show me complete supply chain records, including the source code to your boot loader, drivers, and any firmware that any component on this motherboard is running". And maybe that's ... impossible these days?
I'm the startup CTO, so not the consumer base, but after DHH started posting his thing about cloud exit, I decided to explore this, and I walked the floor of 3 DCs (in SV/TX). People who walk you around the DC don't appear to care about cables, fans or noise etc. If it's longer than 1hr on the floor, they'll put Airpods in and they're done. I've seen cages that look unified, pristine, with love and affection put into cable layout, and I've seen cages that look like a total mess. It's your cage--you only go there if things break.
My last guess is that maybe Oxide rack will end up being sort of what an Apple Macbook is among cheap HP/Acer laptops from Walmart. And it'll be a shiny toy of bold bearded IT dudes who work for GEICO IT by day, but by night they scavenge eBay for used server deals and get excited about the idea of running their own private rack in their basement. They can't afford it at home, but at work they'll want their own Oxide. If you're reading this, do know I know who you are.
Fortune 1000, government. Large organizations that want to own their own hardware, yet want the cloud experience for deploying their software to it. First two customers to have received racks are Idaho National Lab, a Department of Energy laboratory, and a global finances firm. I think that gives a characterization to at least part of the market.
> They won't do this for me?
You are correct that they will do that for you. There are big differences though. From a hardware level, the largest one is that what you'll get in that case are individual 1U servers, in a rack, built from a bunch of reference designs, pieced together from various different organizations. We designed this as a whole rack. From scratch. This has a number of benefits, like for example, we use 80mm fans, at a very low RPM. Our fans draw less power and operate more quietly than the usual fans you'd get in that case. (I know you said people don't care about noise, and it's true that it's not a feature of the product to sell, just an interesting aspect of the design process: we didn't set out to make the rack quiet, it just is, thanks to other decisions that are more meaningful, like cooling efficiently) On the software side, we have written an enormous amount of software from scratch, designed for this specific hardware. Including a control plane, so you can think of the whole rack as a pool of resources, not as individual servers you manage yourself. And since we have done all of this in-house, we can take responsibility for the full quality of the product. If you have a firmware bug, it's not "oh sorry, we'll file a ticket with our firmware vendor and let you know when that's sorted," we will fix it ourselves. Everything is integrated and works together, because we built it for purpose that way, not because we installed a bunch of things from different organizations, ran some test, and said "looks good to me."
It seems they only support up to virtual machines but this is still miles ahead of VMWare, Nutanix, or CISCO ACI to my eyes. This honestly looks simple enough that a developer/DevOps team could manage it. With the other 'hyperconverged' infrastructure offerings you will still need a dedicated team of trained ops professionals to manage it.
Although I am not too sure if DHH will play well with Bryan.
I don't discount that it's a great achievement towards simplicity in the racking approach for rack-scale use cases (having lived through Dell VxRack's nonsense), but let's not kid ourselves that any DevOps team could manage this - do they understand BGP peering? Three phase power requirements? Cooling? ZFS? How about basic maintenance migrations ala Vmotion or DRS? (kidding! they can't do it)
If you don't think Oxide will require an ops team, you may be wish projecting.
A solution as lightweight on operations as Oxide challenges this premise at its core and assuming the capex is not outrageously high, it might suddenly get very attractive.
That seems like the best explanation of the value here I've heard so far! I could definitely see a number of SMBs whose use cases are expensive in AWS/etc and would be a good fit for a few racks in a datacenter, but who don't want the cost of a team to manage them. If this significantly lowers the skill threshold necessary to run servers in a datacenter, I could see that being a huge value prop.
Man I would love to work for a company like that. I don't know why more people don't set their startups like that.
I always find that a very naive point of view, of course I would want to earn a US tech salary while living somewhere in the country side, who wouldn't. But I'm also aware that this is not how the world works in reality, there's different tax systems, different expense costs and we don't live in a global one-market world.
I find the strategy of defining different "zones", like most of the remote first / salary transparency companies much more realistic.
Well the co-founders live in the Silicon Valley area, with their physical HQ being in Emeryville:
For mortgages, you'd need to be looking in the cheaper parts of the Bay, but that still means "dense, boring suburb" as opposed to "crime-ridden slum".
In the case of everyone making the same salary, you could certainly still sabotage someone's success but I don't see how having the same salary makes this type of mentality _more_ likely than at a company with a more typical salary distribution.
My concern with everyone having the same salaries is that, potentially, employees have less motivation to excel in their individual work and are more likely to do the minimum to just stay in good standing with their employer and not get fired. Maybe a company can offset the lack of direct financial motivation with more of a team motivation that the financial success of the company as a whole results in financial reward for the individual or some other way of recognizing individual success inside the company.
I am doubtful this flat salary structure will result in a more successful company overall but I do think it's good to try new things. And yes this has probably been tried a number of times before but maybe not exactly like this. Or maybe some other external variables have changed w.r.t. other attempts in the past and this time it works. The typical salary structures we see in US companies today are the result of a large number of trials and errors and learning.
I totally understand why companies want to pay less though. It's massive cost savings and it makes sense for them to hire for less money.
It's not about what you want, it's about knowing your value. If your work is worth a SF salary then that's what you should be getting.
Moving from Idaho to SF doesn't magically make you more productive. The company knows it's still getting more value from you than what you're being paid. They just want to keep more of that value for themselves whenever possible.
Have some respect for yourself and know your worth
If your entire world consists of staying home and interacting remotely with a company, then you are correct: Location doesn't change anything.
However, moving to a high-energy city with a high density of experienced engineers and tech companies can increase your rate of learning, career advancement, and experience much more rapidly than living in a smaller city. You have to actually branch out and interact with local companies and people, but it does happen.
But this is all beside the point. Hiring is a labor market. Developers who live in SF have more high-paid job options to choose from than someone living in Idaho. As a result, you need to bid more to get them into your company. Hence, the higher salary.
The discussion about cost of living misses this point. The real reason developers from places like SF get paid more is because if you don't pay them wages that are competitive with their local companies, they're just going to walk away and take any number of higher paying jobs they have access to.
How much salary do you think should allocate to base and how much toward any location adjustments, in general?
- https://handbook.gitlab.com/handbook/total-rewards/compensat...
- https://www.checklyhq.com/blog/open-sourcing-our-pay-calcula...
Nevertheless, I agree everyone looks at this problem from their own POV, however it should not be the norm to provide equal compensation for equal work.
In slightly more detail
https://elsajohansson.wordpress.com/2017/09/13/what-does-a-w...
https://elsajohansson.wordpress.com/2022/09/16/the-wage-gap-...
Also, thank you for the links. I'll read this because I'm interested in knowing more about thinking behind this. My instinct is in this direction regarding salaries, but I want to develop more clarity. :)
I wouldn’t. I like cities and center of culture and human activity. And the stats generally show cities growing globally, so I’m not the only one.
While you argue it's naive and cite different tax systems and costs, companies are successfully adopting this approach even in high-cost areas like the Valley. It might be hard to argue that the world doesn't work like that in reality, given that it’s already happening.
While it may not be widespread, it doesn’t mean it’s without merit or unrealistic. After all, remember that today's 'unrealistic' could be tomorrow's norm! It's tempting to think that people in less expensive areas would be the main proponents of a uniform salary, but the reality is nuanced. Preferences are likely influenced by a variety of factors, not just cost of living.
Your later points about the differences between compensation at different company stages are well taken, however it could be difficult to assert how much this dynamic affects preferences given the practice is not limited to early stage companies and equity vests often fail to yield returns.
In addition, your suggestion that only people in poor places want egalitarian salaries, could be seen as disrespectful of other people, because it seems to ignores the totality of an individual while preferring to try to reduce them to simplistic motivations. In that way, it’s also considered abusive. And can also be seen as disrespectful of others experience, and maybe arrogant: "Only people in poor areas want such naive, unrealistic salaries."
Looking deeper, this aspect of your comment, combined with its narrow focus on a single explanation, might be interpreted as your attempt to express your personal frustrations at your own salary performance, or justify and rationalize why you may not be making more. This might occur because you may find it easier to view something you don't have as unrealistic and naive, rather than the result of choices you could change.
In short, while you mention that egalitarian salaries and enthusiastic support of them is naive and unrealistic, it could be argued that the view espoused in your comment is naive and unrealistic because: it lack awareness of complex dynamics; ignores the totality of an individual while preferring to try to reduce them to simplistic motivations, and dismisses real practices as unrealistic, which might also be seen as out of touch. Overall, your views unfortunately could be interpreted as narrow and an expression of personal frustration, instead of a reflection of underlying real dynamics.
To conclude, while it's likely there's some truth to the correlation you propose, it's also likely true that even if some correlation exists, location is not the only factor at play. People may have various salary preferences, independent of their location, just as the value they provide is also independent. Finally, indeed, your view could be expressed more respectfully of others.
Anywho, it's understandable you may have that perspective, given what your background might be. Yet it's always good to remember that you can adapt your view over time, can grow and can include more data to expand that awareness of reality which you value! :)
The egalitarian way where a business can divide the share of revenue that is allocated for salary equally, no matter the role. This makes sense from a philosophy of everybody being a team and contributing equally to the results of the company. It can foster an esprit de corps and and a sense of fairness. On the negative side it could encourage companies to have more burdensome measures of fairness and contribution, and lead to resentment towards colleagues who don't pull their weight.
Then there's the other method, where a value-based salary is allocated to each employee taking into account their experience, ability and effort. Crucially, however, this salary is not adjusted for location. That's the case to which I was speaking, specifically, even tho the type used by 0xide is clearly the egalitarian one.
That's the labor theory of value (see: Smith, Marx), which in theory sounds meritocratic but it can't really be measured or assessed.
In reality compensation either becomes a function of power, social currency and negotiation skills, which is the general norm in professions, or you have an institutionalized, perhaps even democratic process to determine salaries. Both of these variants generate overhead and are only approximations to what anyone would see as fair.
The variant here where everyone gets the same, generous piece of a pie seems refreshingly simple and honest. I would also assume that it attracts the right kind of people, who are intrinsically motivated (at least after the threshold of a very high level of comfort is reached.)
The idea that contribution "can't really be measured" is a cop-out. Contribution can't be measured perfectly but it can be estimated with some accuracy by people who are involved in day-to-day work. "Some accuracy" is really all that's required: as long as contribution is correlated with compensation to some extent, you have a functioning meritocracy.
> The variant here where everyone gets the same, generous piece of a pie seems refreshingly simple and honest.
I bet it works great if you have a small team, are extremely picky about hiring, and quickly fire bad hires. Otherwise it will be awful.
This is handwaving in the extreme. Anyone involved in software development knows that a single line of code can be critical to success or failure, as can a blob of 100k LOC, so product-quantity metrics are of almost no use. The "estimate" you're talking about generally comes down to general "feelings" about who works hard, which have repeatedly been shown to be poor metrics for actual contributions.
> The "estimate" you're talking about generally comes down to general "feelings" about who works hard, which have repeatedly been shown to be poor metrics for actual contributions.
How has this been "shown"? Anyway, you're begging the question that there's some way to determine "actual contributions" that we can compare to "feelings".
If you actually work with a group of people on a daily basis and can't rank order them in terms of usefulness, I find that astonishing. And remember, rankings don't have to be perfect, they just have to more accurate that random.
No, they have to better than both random and "everyone is, on average, and over an extended period of time contributing roughly the same". That's quite a challenge.
Do you go to customers and ask them which features provide the most value to them, and then follow the code back to the people who implemented them? Do you go to the customers who paid the most, and repeat the question to them only?
We're not talking about some award-prize ceremony speech in which we acknowledge that Dmitri and Aneka led the work to get version 8.0 which has been a huge success. We're talking about actual salaries, which are presumably linked in some way to actual sales, and I'm insisting that connecting individual developer efforts to the sales numbers is extraordinarily hard. "More" and "less" are not enough to come up with actual numbers.
This is something virtually all functional companies do when they decide raises. The fact that they don't do it perfectly isn't a huge problem, they just need to be more right than wrong.
> Do you go to customers and ask them which features provide the most value to them, and then follow the code back to the people who implemented them? Do you go to the customers who paid the most, and repeat the question to them only?
Does productivity equal sales? I don't think so. If someone does a good job implementing a feature that doesn't drive sales, that should count toward their productivity. Equally, imagine a task that could be assigned to anyone that drive sales: it doesn't make sense to reward the person who happened to be assigned this task when anyone could have done it.
You're demanding way too much here because you're unwilling to get "handwavy" and instead want some rigorous way to quantify productivity. Instead, embrace subjectivity! Imagine you're in charge and ask yourself questions like:
1. If I need to organize a meeting to address some problem that needs to be solved as soon as possible, who would I invite to the meeting?
2. If someone tells me he plans to quit, how much would I be willing to offer to convince him to stay?
3. If someone quits, how hard are they to replace? In terms of hiring a replacement and/or transferring their responsbilities to someone else.
I suppose there are some workplaces where it's genuinely hard to rank people. But my sense is that they're rare, small, and careful about hiring. Everywhere I've worked, this is not the case and I'm fairly sure this is the norm.
If your company brings in $10M a year, and somebody plans to quit (or has quit), how much will your sales drop (immediately, and over a period of time). That's the answer to how much you can afford to offer them to stay, and that's how much you should offer their replacement. Suppose that says drop by $1M and you can be satisfied that the drop is 100% a consequence of the departed (or soon to depart) developer - that's how much value they bring to the company, and by the logic of capitalism (which I don't play by, btw), you should pay them some amount less than that.
The problem is: you can't determine the value before they quit, and you can't be sure that their replacement will provide that value after they are hired.
Look, I understand that in an organization of any size, there are likely to be slackers that feel like a deadweight, and others who feel like the contribute far more than the average.
The question is: does tying salary to this perception actually bring the benefits you think it does (which are intimately connected to the notion of incentive) ? There's some good evidence that for developers and other "head-based" employees, it does not, and that flat pay scales create an environment in which you get different kinds of benefits.
I've worked exclusively in a distributed FLOSS project for the last quarter century, so in many respects, I'm not well positioned to talk about what happens inside traditional corporations.
Even if you actually do measure and agree on metrics, then the measurement can easily become the goal for those who are not intrinsically motivated. Work ethic can't be taught by dangling carrots in front of people, because acquiring the carrot becomes the goal instead of moving the cart. This can be detrimental in a highly collaborative workplace.
Having a flat, generous salary might solve this problem, because you filter out carrot hunters and get cart movers.
> I bet it works great if you have a small team, are extremely picky about hiring, and quickly fire bad hires. Otherwise it will be awful.
Finding the right people to work with is difficult regardless. The same worker can be miserable in one place and flourish in another.
Besides, LTV as a theory is meant to be a description of the world as Smith, Marx, etc. see it, not a prescription for how things should be done.
But the underlying belief of paying someone according to their effort, is very much based on the same premise.
What I'm saying is that nobody is _really_ paid according to their effort, experience etc. because those things cannot be reasonably measured.
The typical process of determining compensation is based on negotiation and power. In some places the process is more democratized and rules based. Both of these are only to some degree related to actual effort, experience and so on. This discrepancy becomes larger the more people are involved as well.
The skin-in-the-game bit can be promotions, equity, stock compensation, etc.
I've worked at two companies where everyone got the same base salary, but the variation came in equity.
If ya'll are looking for a remote security engineer from Europe hit me up. :)
For those numbers I'll walk the servers into the customers myself.
With this in mind, it's just a good salary. Until they have competition squeezing them on margins it looks like an OK approach. At some point their board most likely won't agree with paying some people below market rates for important roles and other more fungible roles paying 2-3x market rates but while they can keep doing it, it's great free marketing.
So the extremely high salary is still a net positive compared to EU
Plenty of that in London unfortunately and our salaries are half.
>Healthcare
And not so much of that... despite paying tax for it anyway.
Income tax: 34% total (top marginal rate of 39%).
Leaving $133k.
Other taxes (GST~VAT 15%, city rates, high petrol excise, etcetera) will easily take another $10k. Interest on your home is not deductible (NZ has very few deductables). People earning six figures will often pay for private health insurance and medical fees on top of the socialised healthcare - maybe another few thousand.
homes, etc is true though and we just need to build more inventory IMO. but there are tons of areas with affordable homes, they just aren't near the big cities like NYC or SF or LA
I took a base-pay cut to work at Oxide. Zero regrets.
I doubt my Aspen vacation home developer will accept payment in startup stock.
I live in flyover country, and there are no shortage of talented people here, but I wouldn't dispute that the per-capita software developer talent in SF is going to be higher.
But for senior engineers at these companies, 200k is pretty competitive with the base salaries offered at those places
I meant 200k is just base salary at Oxide. It does not include equity
Open AI just had a tender offer for employees this month. It's not public. Clearly the equity is worth something even though the company is not public.
Some companies facilitate secondary transactions for their employees.
There are ways many employee friendly companies provide early liquidity without IPO.
Yes it is somewhat of a stumbling block, but I'm betting they are getting extremely high quality employees who are doing it for the passion and not just for the money.
All of that said: thank you for keeping your fingers crossed. ;)
> Cloud computing is the future of all computing infrastructure.
Oh god no!
Anywhere I can donate to have this not happen? I want my computing on premise, preferably under my table.
But well, if it _HAS_ to happen, 0xide is probably a lesser evil.
Let's start with their beliefs: 1. Cloud computing is the future of all computing infrastructure.
Yeah, no. There's still gonna be things like mobile phones which will do some stuff locally. And probably also desktops. And probably also on-prem servers.
Which brings us to their belief no. 2: The computer that runs the cloud should be able to be purchased and not merely rented.
If you buy a computer and run virtualized loads on it, then you're not doing cloud computing. As simple as that.
Basically, as others have commented already, this is just something like a blade server system or a hyperconverged system like for example Nutanix is offering. The only real difference seems to be the open source approach.