Building big systems with remote hardware teams | Hacker News Reader

Building big systems with remote hardware teams

oxide.computer

105 pointsirsagent20 comments

> All of that to say, the cost of building small boards is really low. A single prototype run that saves a "real" board re-spin pays for itself immediately. Enabling software development before "real" hardware lands pays for itself immediately. Even when things don’t work out, the cost of failure is low; we lost a few hundred dollars and some engineering time, but learned something in the process.

Yes. This is the absolutely central aspect of modern software development, and it's good to see the same being applied to hardware through cost and iteration time reductions.

The modularity is interesting as well: it's basically building their own Arduino-like system of pieces.

This post was a great read and it's exciting to see companies prove that hardware design can be done remotely. The real gem here though is the silkscreen on those boards, that was great to explore and laugh!

Listen to Oxide's podcast "Oxide & Friends", it's awesome if you're in this space of datacenters, servers, networking etc.

Last episode was about doing their own networking, and it's not some whitelabel $hit, they go to the firmware level and are trying to open up stuff like a server's BMC. (jesus christ those things are horrendous.)

[1]: https://www.techopedia.com/definition/15941/baseboard-manage...

Love their hardware prototyping strategy. Unit testing makes so much sense, but I've rarely seen it done in hardware.

I'm curious how big their hardware team is?

The official hardware team is about 10 people, but there's another 5-10 people that somewhat overlap.

Several of the boards discussed in the post were designed by people that are nominally on the embedded team, e.g. I designed the Gimletlet NIC and Root-of-Trust-Carrier-Carrier.

Author here. One thing I really enjoy about this team is that we're trying to avoid silos and build a cohesive team that can flex when needed. As Matt mentioned, a number of the boards discussed are a result of someone jumping in and helping where there was a need. The Gemini bring-up board was done before we even had an "official" hardware team.

Pretty cool how fast prototyping PCBs is. In a few years maybe you can do that with chips as well, with FPGAs you already can. I bet in a few years the have costume RISC-V in a few days for a few 1000$ as well. Next step, in house mini fab.

Wafters spend something like 3 months in continuous processing in a fab. Considering the extreme difficulty of the physics of the task and the scale of investment into fabs where small improvements in speed would be worth billions, I would be very skeptical that any low hanging fruit remain for dramatic shortcuts of the process.

I think the actual process time is a lot shorter than that, but the dynamics of trying to (1) keep the fab 100% busy and (2) satisfy the most lucrative customers first mean that your job gets done at some point in a 3 month window.

I am not an EE but my understanding is the time varies a bit based on complexity of the design (ie how many layers) but that even if you're willing to pay a multiple of the basic price the shortest you can expect is 2 months. There's physical steps that simply cannot be sped up more than partial inventory simply sitting around in a queue. Googling real quick agrees with that impression.

There's just no way the current approach will get to the days turn around the top comment was betting will happen in a few years. If there was anything even remotely promising that with commercialization in a few years we'd all have heard about it loud and clear.

Likewise, the support equipment, the chemicals involved. There's just no chance of current fab tech becoming a 3d printer like table top device. That would require a revolution even bigger than the original chip revolution.

Please dont assume, if you dont know. Growing oxide, doping silicon, depositing metal, grinding all take substantial time. Even if you do a magical „no logistics everything in one place machine fab“ it would still be months of non-stop operation to do 200-300 process steps required.

I didn't know that, thanks for the information.

The test boards remind me of this old article from Sparkfun, about how they use pogo pins to build custom testing jigs to verify products.

https://www.sparkfun.com/tutorials/138

How does the hardware talk?

If it's not via software, I'm willing to bet the hardware is not remote. Or if not exactly software, it's contractual signaling, which we can model and understand and MITM in software.

If it is by software, we can model these problems just like software. Known problems with known solutions. With the understanding that individual systems may incur side effects (like hardware bugs or . . . UI!).

Most of this is not about hardware and a lot of it is unproved conjecture. There is also a smattering of hardware-specific knowledge (this wears out that way!) that is not about building large systems, but about problems in the small.

It's not the hardware which is "remote" but the team, i.e. a set of people working from home in different places rather than lab workbenches.

I think the parent has a point as the author makes a big deal of how this hardware supports their remote work.

It’s clear to me oxide are having fun, and while you could do some of this in software, you clearly couldn’t do that without some hardware. Could they have purchased the hardware instead? Probably. Would it have been as cost-effective or fun? Probably not cost effective, without shifting their staffing. Does anyone have fun purchasing OTS development hardware? You’re just gambling on what systems and complexity in their own stuff they’ve really tested.

Changing staffing is hard, expensive and painful. Repurposing apparently excess hardware development folks to fill the gap here is interesting, not the way hardware is “usually done” and seems to be working.

OTS development hardware is fine for very initial prototyping/derisking phase, but as your product evolves you need your development hardware to match the hardware in the field more closely.

Yes but all hardware companies do that, and that is not what the author seems to be discussing. The article seems to me to focus on the unique nature of making hardware tools to support remote work.

>Could they have purchased the hardware instead? Probably

Uh, from the article:

>We use many off-the-shelf development boards and even provide support for a number of these boards in Hubris. These are great for many use-cases, but less great when we are trying to model specific circuits or subsystems of our product designs. Second, we have numerous examples of custom boards that were built simply because we could find no useful OTS (off-the-shelf) alternative.

Yes but here the author mixes tool-making and product de-risking. I was talking about tool-making which seemed to be the unique part of the article. Everyone designing hardware products designs some hardware.

You can almost always make tools and test equipment with general-purpose hardware. There are companies that design hardware intended to be used like that. People tend to (understandably) try to drag problems into domains where they are comfortable solving them. In this case they tended to draw solutions from hardware but a software team probably would have solved them a different way.

Reply on news.ycombinator.com