Oxide at Home: Propolis Says Hello
artemis.sh
artemis.sh
If you haven't heard you should check out their podcast, "On the Metal [0]" It's a truly a gift especially if you are an elder millenial.
Recommendations technical:
- Twitter Space 12/13/2021 -- The Pragmatism of Hubris
- Twitter Space 12/6/2021 -- Tales from the Bringup Lab
- Twitter Space 11/29/2021 -- The Sidecar Switch
Historical:
- Docker, Inc., an Early Epitaph
- Twitter Space 7/5/2021 -- NeXT, Objective-C, and contrasting histories
- Twitter Space 5/31/2021 -- Silicon Cowboys
- Twitter Space 8/16/2021 -- The Showstopper Show
All of them are worth watching.
On one hand I'm excited at the new content but a bit weird they way they stealth shutdown the podcast and started the twitter spaces thing up without saying anything or updating their homepage. Even their twitter feed only mentions the twitter space a couple of times (and doesn't appear to link to the archive).
Oxide if you're listening, please publish these on the podcast feed to make them more accessible and followable.
I read a note quickly about Oxide and Twitter Spaces a while ago, but as I read the keyword Twitter with a mindset of "good for short notes and links to external sources" as I wasn't at all aware of voice capabilities, might be shortfall of using Twitter in the browser..
Thus I now have lots of interesting material to enjoy.
It's not Openstack. It's not VMware. It's not kubernetes. It's not proxmox. It's not Xen. It's not Anthos. It's not GCDE. It's not Outposts.
So who and what is it for? Where is the use case that none of these other products fit the bill?
Especially for an on premise use case.
The business is fairly straightforward: we sell computers, a rack at a time. You as a customer can buy a rack, and put it in your data center. The rack offers an in-browser management console, built on top of an API you can use too. You use these tools to set up virtual machines. You can then use those VMs however you want. You get the cloud deployment model but with the "I buy the servers" ownership model.
There's a few different advantages depending on how you want to look at it.
Starting from a rack as the smallest unit rather than 1U brings a lot of advantages, but there aren't really vendors currently selling these sorts of things, instead "the hyperscalers" have internal teams building stuff like this. There are a lot of organizations who want hyperscale style servers but aren't going to start a division to begin making them themselves.
Another advantage is that everything is designed to work with the rest of it: you (or the OEM you're buying from) are not cobbling together a bunch of hardware, firmware, and software solutions from disparate vendors and hoping the whole thing works. Think "Apple," or "Sun," rather than "IBM PC Compatible." This is easier for users, as well as allows us to build systems we believe are more reliable.
There's also smaller things, like "as much as possible everything is open source/free software," which matters to some folks (and allows for interesting things like the above blog post to happen!) and is less important to others.
How does this differ from what large players like Dell are offering under the "hyperconverged" moniker. For example, Dell's Vxrail[0] appears (from marketing speak, anyway) to be a single rack with integrated networking and storage that you can ask to "just start a vm".
[0]: https://www.dell.com/en-us/dt/converged-infrastructure/vxrai...
"hyperconverged" is a term used by VMware to describe a virtualized all-in-one platform. You get compute, storage, and networking, all virtualized as one appliance rather than as individual ones. VxRail is basically Dell EMC's implementation of this idea: you get one of their servers, vSAN and vSphere all set up and ready to go.
"hyperscale infrastructure" describes an approach to designing servers to begin with. A lot of folks moved toward commodity hardware in the datacenter a decade or two ago. And then you get more and more of them. The hyperscale approach is kind of top-down as opposed to that bottom-up style: how would we design a data center, not just a server. Don't build one server and then stick thousands of them in a building; think about how to build a building full of servers. This is more of an adjective, like RESTful, rather than a standard, like HTTP 1.1. That being said, the Open Compute Project does exist, but I still think it's closer to a way of thinking about things than a spec.
Okay, so all of that is still a bit fuzzy. But it's enough background to start to compare and contrast, so hopefully it makes a bit more sense.
The first difference is the physical construction of the hardware itself. If you buy VxRail, you're still buying 1U or 2U at a time. With Oxide, you're buying an entire rack. The rack isn't built in such a way that you can just pull out a sled and shove it into another rack; the whole thing is built in a cohesive way. This means that not every organization will want to own Oxide; if you don't have a full rack of servers yet, you don't need something like we offer. But if you're big enough, there's advantages to designing for that scale from the start. This is also what I meant by there not being a place to buy these things; other vendors will sell you a rack, but it's made up of 1U or 2U servers, not designed as a cohesive whole, but as a collection of individual parts. The organizations that are doing it this way are building for themselves, and don't sell their hardware to other organizations. This is also one way in which, in a sense, Oxide and VxRail are similar: you're buying a full implementation of an idea from a vendor. Just the ideas are at different scales.
The other side would be software, which of course is tied into the hardware. With VxRail, you're getting the full suite of software from VMware. You may love that, you may hate it, but it's what you're getting. With Oxide, you're getting our own software stack, which the article is about the details of. You may love that, you may hate it, but it's what you're getting :). That being said, I haven't actually used a full enterprise implementation of the VMware stack, so I don't know to what degree you can mess with things, but our management software is built on top of an API that we offer to customers too, so you can build your own whatever on top of that if you'd like. Another thing here is that, well... the VMware stack is not open source. All our software will be. That may or may not matter to you.
The last bit about software though, is I think a bit more interesting: even though you are buying a full solution from Dell EMC, you're also sort of not. That is, Dell and VMware are two different organizations. Yes, part of what you're getting is that they say they have pre-tested everything in the factory to make sure it all works together well, but at the end of the day, it's still integrating two different organizations' (and probably more) software together. With Oxide, because we're building the whole thing, we can not only make sure things work well together, but really take responsibility for that. We can build deep integrations across the entire stack, and make sure that it not only works well, but is debug-able. Dell EMC isn't building the hypervisor and VMware isn't writing the firmware. Oxide is writing all of it. We think this really matters for both reliability and efficiency reasons.
So... yeah. That's a summary, even though it's already pretty long. Does that all help contextualize the two?
I also love that they continued to bet on Illumos and am looking forward to the continued growth and development in the Illumos space.
0xide is one of the few really interesting new tech companies out there.
I have to admit some bias, as I was involved with a company offering a “poor mans vBlock” around 2010. We didn’t grow fast large but we never once lost a deal against commodity hardware vendors. They were easy to beat.
Also in my experience, a fabulous candidate is sometimes only available for a very short time window (they either have become available due to unforeseen circumstances, or they are snapped up by a faster mover).
Is six weeks fast in your opinion?
The places I'd like to work seem to move slower and require more preparation. For instance Oxide has applicants put together a lot of written work and another company has a seemingly easy take-home project but which requires unfamiliar (to me) setup that I haven't had time to tackle. Then there are the faang-style interviews with loops scheduled perhaps weeks in the future.
I may send another round of applications soon, but I'm going to be more selective so I can manage the process better.
In the first case, you would bill something and get a salary out of it. If you bill x, a) your income is in dollars rather than local currency, which from experience is not great, b) you cost less to the company than US employees who cost x times k, c) you have to pay your own overhead before making it a salary. So if you bill x, your salary is less than x, and you still have to pay taxes on that salary.
If GEO or other similar arrangement, you're usually paid in local currency, presumably an amount corresponding to x at a given date, and that has extra overhead for the employer different from what a normal employee would cost.
Either way, "everyone is paid the same, regardless of location" doesn't clarify much. Thus my original question.
Currently we are using a remote hiring platform with local legal entities, so you'll get paid in local currency and receive local benefits. I'm based in the UK, I don't have the overhead of medical insurance, but the platform itself has costs and I'm not sure how much the costs of a UK employee compare to other countries.
But, we don't have many international employees, so the setup might change in the future or depending on the individual. When I first started, I briefly worked part-time as an independent contractor.
What would the hosting story look like now if 8(?) years ago 25% of servers had adopted SmartOS?
I never used it in production anywhere, admittedly. I also never got a chance to try out Triton. I'm on the fence about whether or not I keep my SmartOS hosts in my home network now that Illumos is a second-class citizen when it comes to OpenZFS.
The case with OpenZFS does worry me as well. I fear the developers start slowly introducing linuxisms thus sacrificing portability and stability for the great penguin.
I would love to hear more about the replacement of VMware experience and any other Triton details good/bad.
Generally there hasn't been anything game breaking and our biggest issues have been with running out of logging space for the core services (went with too low capacity disks in the beginning, doh) and one failed upgrade which didn't even take down the whole cluster during fixing. Our clients can now also provision their own VMs with the Triton API unlike with VMware which required admins to do it. Bhyve like KVM before has also been rock solid in everything we run from simple web servers to kubernetes clusters.
For issues we've found the Joyent/SmartOS IRC channels to be excellent and they have helped us tremendously in debugging and fixing things. It's the best support I've encountered for a FOSS product by far and one of the biggest reasons I'm such an illumos advocate now.
I'm also looking forward to further testing LinuxCN (https://github.com/joyent/linux-live/tree/linuxcn) on Triton in the near future!
Are you running Manta (https://github.com/joyent/manta) for anything? If so, is that meeting you needs for object storage?
Not sure about linuxCN either. I would rather run !linux on bare metal as much as possible, and having purged most of the penguins from infrastructure it would feel a bit weird to go immediately back ;)
In addition, we also have SPARC hardware running on OpenBSD due great hardware support, including ldoms! Of course would be nice to run illumos on them too, but alas...
What problem does Oxide solve exactly?
The point of a startup is to disrupt the current market. And they are trying. Lets see how they do.
Does it matter? Oxide is building the whole stack - their own hardware with their own firmware to run their own OS with their own virtualization layer. They don't need support for arbitrary devices, because they control the hardware.
tl;dr: Smaller players are able to do things that larger players can't -- which isn't to minimize how challenging it currently is!
Moore's Law has been dead for a while. Getting "performance" now requires design and architecture again rather than just sitting back for 18 months and letting Moore's Law kill your competitor.
The big problem right now is that custom chip hardware is still too stupidly expensive because of EDA software. Fab runs are sub $50K, but EDA software is greater than 100K per seat and goes up rapidly from that.
Yes, actually, you do.
The "interesting" bits in chip design aren't the digital parts--the interesting bits are all analog.
A RISC core is an undergraduate exercise in digital design and synthesis in any HDL--even just straight Verilog or VHDL. It's a boring exercise for anyone with a bit of industry experience as we have infinite and cheap digital transistors. (This is part of the reason I regard RISC-V as a bit interesting but not that exciting. It's fine, but the "RISC" part isn't where we needed innovation and standardization--we needed that in the peripherals.)
However, the interfaces are where things break down. Most communication is now wireless (WiFi, BLE, NB-IoT) and that's all RF (radio frequency) analog. Interfacing generally requires analog to digital systems (ADCs and DACs) and those are, obviously, analog. Even high-speed serial stuff requires signal integrity and termination systems--all of that requires parasitic extraction for modeling--yet more analog. And MEMS are even worse as they require mechanical modeling inside your analog simulation.
If your system needs to run on a coin cell battery, that's genuinely low power and you are optimizing even the digital bits in the analog domain in order to cut your energy consumption. This means that nominally "digital" blocks like clocks and clock trees now become tradeoffs in the analog space. How does your debugging unit work when the chip is in sleep?--most vendors just punt and turn the chip completely on when debugging but that screws up your ability to take power measurements. And many of your purely digital blocks now have "power on/power off" behavior that you need to model when your chip switches from active to sleep to hibernate.
All this is why I roll my eyes every time some group implements "design initiatives" for "digital" VLSI design--"digital" VLSI is "mostly solved" and has been for years (what people behind these initiatives are really complaining about is that good VLSI designers are expensive--not that digital VLSI design is difficult). The key point is analog design (even and especially for high performance digital) with simulation modeling along with parasitic extraction being the blockers. Until one of these "design initiatives" attacks the analog parasitic extraction and modeling, they're just hot air. (Of course, you can turn that statement around and say that someone attacking analog parasitic extraction means they are VERY serious and VERY interesting.)
Having "infinite and cheap" transistors is what makes hardware design not boring. It means designs in the digital domain are now just as complex as the largest software systems we work with, while still being mission-critical for obvious reasons (if the floating point division unit you etched into your latest batch of chips is buggy and getting totally wrong results, you can't exactly ship a software bugfix to billions of chips in the field). This is exactly where we would expect shifting to higher-level languages to be quite worthwhile. Simple RISC cores are neither here nor there; practical multicore, superscalar, vector, DSP, AI etc. etc. is going to be a lot more complex than that.
Complicated analog stuff can hopefully be abstracted out as self-contained modules shipped as 'IP blocks', including the ADC and DAC components.
Cost competitive as it may be, it doesn't mean that it hits the price point for a home lab, sadly. One of the (many) advantages of an open source stack is allowing people to do these kinds of experiments on their own; looking forward to getting our schematics out there too!
I know several people with 2x 240V 32A 3-phase in their garage, that's 20+ kW at any reasonable power factor. But a 15 kW cooler that would work in summer would annoy the hell out of any neighbours living closer than a mile.
It would be nice to have a known pricepoint from a cloud provider which once exceed you ask the question: "Should we buy a rack and COLO it?" Even if the answer is "no" it's still good to have that option.
---
The thing is: Datacenter technology has moved on from 2011 (when I was getting into Datacenters), but only for the big companies. (Google, Facebook, Netflix); I think Oxide is bringing the benefits of a "hyperscale" deployment to "normal" (IE; single/double-digit rack) customers.
Some of those things such as much more efficient DC converters, so not every machine needs to do it's own AC/DC conversion.
That's not surprising, you're basically paying for scalability. An idle box doesn't even necessarily "waste" all that much energy if it's truly idle, since "deep" power-saving states are used pretty much everywhere these days.
Each stick of DDR4 is going to consume on the order of 1.2w (idle CPUs can theoretically go lower than this).
I’d rather shut a whole machine down than go to the effort of offlining individual DIMMs, since the consumption is so low and quite static.
They’re not reasonably equivalent. But I don’t doubt that Amazon is laughing to the bank still.
Their gear is targeted at way larger-scale than I'll ever get to use (what with the size of environments I work in). What I hear about their attitudes re: firmware, for example, makes me wish that I could have their gear instead of the iDRAC's, PERCs, and other closed-source roach-motel hardware I'm stuck with.
I'm young enough that I just missed the era of computers that Oxide evokes. I put in a couple DEC Alpha-based machines in the late 90s and got a glimpse of what it might be like to have a vendor who provides a completely integrated hardware/software stack and "ecosystem". I'm sure there was operational advantage to being a "DEC shop" or a "Sun shop". The PC market crushed that old school model by wringing out the margin necessary to make that kind of company work. I'd love to see Oxide make a go of it, though.
Who knows? In twenty years, we could see something cool come out of this, like better U-Boot tooling. Or maybe they'll be purchased by Oracle, which would if nothing else be funny this time.
Everything else "new" seems to be a rehash of the IBM PC (my "server" has an ISA-- ahem-- "LPC" bus... >sigh<). It's so refreshing to see something actually new.
The same goes with software and firmware. Any "new" systems software the last 10 years seems to be thin management veneers over real technologies like the Linux kernel, KVM and containers, GNU userland, etc. And it all ends up running on the same cruddy BMC's, "lights-out" controllers, embedded RAID controllers, etc.
I get a little bit of excitement at ARM-based server platforms (and RISC-V, for that matter) but everything there seems to be at even less of an "enterprise" level (from a reliability, serviceability, and management perspective) than the PC-based servers I already loathe.
Oracle will likely be very interested in Oxide, but I suspect Bryan Cantrill would do everything in his power to prevent that happening. He's seen the lawn mower in action before and knows not to anthropomorphize it :)
I'm very sad about what Silicon Valley has become. We speak of "tech companies" but they mostly no longer exist. What are the big names in Silicon Valley now? Advertising companies, for the most part. A movie company. Online shopping. Social whatevers. None of these companies sell tech products, they are not tech companies. Sure, they use tech internally but so do law offices and supermarkets, those aren't tech companies either.
I miss the Silicon Valley of actual tech companies. Sun, SGI, HP (the actual tech HP of back then), etc. Apple survives, but focused on consumer-level stuff which I don't find interesting. Oracle is around, but they were always more of a lawyer shop than a tech company. Real hardcode tech companies, do any exist anymore?
Oxide is such fresh air, exciting!
Every week or so I'm about to send an application, I really want to work there. My partner would kill me though, so I haven't. (They have a flat pay scale that, when living in Silicon Valley, would make it very difficult to support a family.. so I'm stuck cheering from the sidelines.)
But my secret hope his to eventually get an Oxide branded developer laptop. The design language alone would make it worth it. But not holding my breath as that clearly not their priority.
I wonder how hard it would be to port to FreeBSD.