syntheticgate

Our thesis is that it's both a hardware and software problem! Which, as steve said, is why we're doing both. We can look at designs across the whole stack, from chips chosen, FPGA implementations, hubris management and control plane software to accomplish this goal. By doing these together, we can accomplish fundamentally different designs and tighter integration than any one of those multi-company silos can even if they tried to work together.

Yes it certainly can be done, but there's a cost and design complexity with doing that too. I did a quick count of gimlet (our server sled's) power rails and got to over 26 different power domains, and I probably missed a few in my quick scan! It's unclear to me if the efficiency gains from re-doing these with something more exotic to go from 54V would make enough of a difference to justify doing so, and we'd still end up with some stuff like the SVI2/3 controllers needing an intermediate rail (or have to go design those ourselves too) and some analog rails needing LDOs for noise rejection reasons etc. As mentioned before, creepage and clearance at higher voltages cascades into layout complexity and pain if you have to run it everywhere on the board: but for the same reasons we're talking about this, we can't very well do a 54V:1V conversion in the back of the sled and run it all the way to the front- losses, noise etc.

As with all things engineering is a series of tradeoffs and right now, an IBC from 54V to 12V has been a reasonable design point for us.

In Oxide's design, we do have a 54V DC busbar so that's what the rectifiers put out, and runs vertically up and down the back of the rack. The power connection into each of the cubbies for the sleds, and the power into the sidecar switches connect to this bus bar at 54V. Each of these assemblies has an intermediate bus converter IBC that does the 54->12V conversion on board and 12 and other lower rails are used for the various supplies required.

We do run the 54V to our fans (in both sleds and switches) without additional DC-DC conversion as those can be fairly power hungry and we can buy reliable fans that are rated for this voltage.

Our sleds actually don't connect directly to the bus bar to help mitigate some of the "oops" factor as they're going to be potentially mated and unmated buy customers as they reconfigure, upgrade or support things. The sled cubbies are wired to the bus-bar and support hot insertion of the sleds. And yes, while possible, an in-field replacement of the bus bar wouldn't be fun, but in our design it's a big copper bar hidden away so the risk of damage or dropping stuff into is minimal.

So our 12V IBC design gets us into more normal range for commodity point-of-load supplies, and balances the losses due to higher current at 12V vs the complexity of dealing with 54V all over the boards. For the AMD parts, we also have to have supplies that deal with SVI2 or SVI3 where the part itself can adjust its voltage at run-time for efficiency. These are pretty complicated devices (like the RAA228218) that we're happy to not have to design ourselves and they have expected operating envelopes for their supply-side rails that don't work at 54V.

At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.

Huge fan of nvc, can't recommend it highly enough! It is fast, has excellent language support and issue response on github is amazingly fast even though this is a hobby project. We're using it every day at work.

I'll second the Kester recommendation, the stuff is a bit expensive but if you've used bad solder you know it's worth it! I'm using Kester 275 on the bench and have been very happy with it, and as it's a true no-clean flux, it's a bit less messy than Kester 44 without much performance difference (lead free vs lead differences are real still- Lead-free is still not as easy to work with, but the world went lead free so my bench did too). You can find some comparisons here (since kester's website is down currently for some reason) https://www.jameco.com/Jameco/Products/ProdDS/2188246.pdf

lol, I'm working on an FMC-FPGA interface at work right now and discovered this same chipselect behavior.

I believe this is referring to a little board with an STM32G031 part that exposes a PMOD i2c interface and we use it for programming serial numbers into FRUID EEPROMs on the manufacturing line. It runs hubris, can be controlled by humility via SWD, and has just enough space to do this function. We have other dev boards using larger STM32 parts and didn't want to consume those for our manufacturing line stations.

Given the size of the part on here, and the limited I/O, it's not very useful outside of this use-case but works great here.

We ship you the whole rack you bought, either to your own on-prem datacenter or your preferred colo partner etc. Once on-site, assuming the infrastructure is already in place, it's ~3hrs to uncrate, connect power and network, and do the site-specific install configuration (like DNS, time servers, identity etc etc).

I'd suggest design complexity has scaled at a faster rate than the autorouting tech. In a closer-to-DC world years ago, optimal routing was nice but often not a requirement. In today's high-speed designs, proper routing can be the difference between working and not working, and there is a constant battle of constraints and trade-offs that humans can deal with more effectively. Attempting to constrain a complex design such that an auto-router can be successful can often result in an over-constrained problem which then makes the auto-router unsuccessful.

More focus on hybrid routing tools where there is some automation going on while a user routes things is typically were you get much more useful help in modern ECAD.

Not really, at least not apples-apples for something like a Pi.

There are a couple of different aspects to this, one is that soft-logic it typically slower than hard-logic so you just can't get comparable frequencies out of a soft implementation. For datapath designs, this is typically solved by going wider, but that isn't quite as helpful or practical for all aspects of a processor implemented in soft logic.

If you look at the specs for this softcore processor, they have much less performance than a Pi, even when you're using some of the biggest and more $$ families of FPGAs: https://www.xilinx.com/products/design-tools/microblaze.html....

I'd say that is on-par with similar complexity soft-core CPUs from other vendors or even open-source ones.

With respect to the design transparency, it kind of depends on how much you care about the black-box compilers required to use a lot of these advanced chips. You can feed open-source RTL into them, but there's still a proprietary black-box compiler/fitter/place-route etc for a lot of these.

There's some work toward open toolchains from yosys and https://f4pga.org/, but none of the big FPGA companies seem very bought-in or willing to help in big ways, so it's been a community best-effort, and for some of the fancier devices, you still have to use the proprietary tools to build bitstreams.

Author here. One thing I really enjoy about this team is that we're trying to avoid silos and build a cohesive team that can flex when needed. As Matt mentioned, a number of the boards discussed are a result of someone jumping in and helping where there was a need. The Gemini bring-up board was done before we even had an "official" hardware team.