Oxide and Friends – Rack-scale Networking [audio]
youtube.com
youtube.com
This conversation goes into the details about making all of this work. Along the way, we hit on pushing a cycle-accurate simulator to its limits; developing a new P4 compiler that has a Rust target; using that compiler to then develop an accurate hardware-virtualized model of the ASIC; developing a new routing protocol (delay driven multipathing); and how all of this adds to up solving some gnarly problems like microcongestion and flexible topology construction.
[0] https://share.transistor.fm/s/c80a79d8
[1] https://share.transistor.fm/s/65a10522
[2] https://p4.org/
I say this not to detract from your work but because I’d genuinely like to see companies opting into self-hosted hardware as Oxide envisions. The centralization of all internet services into three big cloud vendors does not bode well for society.
given his skills and experience compared to anyone I've worked with (at least at the time that I worked with them) I would trust what Bryan puts his weight behind over the internet's techno-fetish of the month, any day.
you must always work on things that have value to you, even if those things are unknown or unpopular. not because they are unknown or unpopular, but because they are the right solution for you.
the amount of homogeny that I see today staggers me. people don't seem to be solving their actual problems, but instead are twisting their problems so that the commonly deployed solutions fit.
Oxide seem to be developing solutions for people that have problems which the common solutions can not address, no matter how much you twist them.
In terms of Oxide: I can assure you that we are shipping the minimum viable product -- the challenge is that the minimum viable product is very (very!) large. (And indeed, the projections we made in our initial investor deck were far more accurate than they had any business being!)
Glad to hear that you're cheering us on, and thanks again for the kind words!
PS: I would really love an episode about Sun deciding to go to bed with AT&T and merge and BSD and System V. That seem to cast a long shadow and must have left some scars.
I get that this is something you just hit record on while chatting on a work laptop or something, but the 's' sounds in your audio track especially are so high pitched it's unbearable to listen to for me.
I will say that this recording does sound better than a much earlier one I tried a while back. So if you did change your mic at some point, it probably helped quite a bit.
I just listened to the first couple minutes, but generally quite a few of your 's' sounds trigger the nails on chalkboard type reaction(not that bad though, but same type of reaction). Hope this makes sense to you. Im a couple decades younger than you, so could be I'm hearing some high frequencies you couldn't hear.
Keep up the great work at Oxide!
The old “pencil trick” can be highly effective, even with good mics: https://urm.academy/death-to-sibilance/
Disclaimer: I work for Cisco.
If you allow me to query on a different oxide topic: I noticed that you provide a certificate to each HSM in every system that is signed by an Oxide Computing root cert. This is pretty neat because, combined with other security features, it enables clients to validate that the running firmware came from you. Have you considered publishing a certificate transparency log of all certificates generated for hardware root of trust HSMs? This would enable customers to ensure they are not using firmware signed using a compromised root certificate. https://eprint.iacr.org/2022/981
As someone deeply involved with the P4 community and contributor to the P4 compiler it is great to see someone being bullish on the language. Have been following your work with interest. :)
I see that you are developing your own compiler. If you have complaints or suggestions please feel free to reach out to p4-design@p4.org. The language design group is meeting monthly and is always interested in hearing grievances of consumers of the language.
As a side-note, we are currently developing tools to ensure that P4 programs are compiled and executed correctly. Tools include packet-test generators, fuzzers, and general-purpose verification frameworks. Because the language is so restricted you can do really powerful validation. A concrete example is Google using P4 as a specification language for their network devices[0]. If that is something that interests you, shoot me a message (fruffy@nyu.edu).
I'd say... there's a non-zero chance this work alone could revive it, show the technologies worth.
There's a ton of value to having an understandable clear networking stack that you can see & work on, and the podcast is totally correct that vendors almost never offer clarity or possibility. Netops crosses fingers, flips a switch, & sees what happens. By contrast, this is a refreshing take on being responsible for your shit, in a way little else does.
But it's still unclear that embedding much more complex computing in your network switch has real value. Having a far more configurable P4 capability under the belt isn't a little step towards ownership, it's a huge step towards software-defining the network pipelines.
This seems unrelated, but I somewhat look at systems fabrics - smaller scale & faster connections - & see a lot of complexity about figuring what to send where. The idea of embedding more intelligence in the connective fabric has huge alure. But it's unclear who or when someone will truly leverage more flexible switching/interconnect to great effect.
As for Tofino 2, I hit on this earlier when Intel first announced it, but fortunately Tofino 2 is still being supported: the team has been very candid and transparent with us (which we very much appreciate!) and Intel has been both formal and explicit about their ongoing support for Tofino 2. We remain -- for all of the reasons we went into in this discussion -- very bullish on P4, and we believe that we will be able to show its unique value in the Oxide rack. A long way of saying: we are absolutely with you on the non-zero chance of showing the technology's worth!
So it was _really_ interesting to listen to this podcast live and think about truly high performance networking. When your requests and responses are all within one rack on your data center, the bandwidth, latency and throughput bottlenecks are MUCH more likely to be your code and hardware. So Oxide has to think about all these high-performance use-cases that I've never once had to stop and think about.
This episode felt like peeking behind the curtain into forbidden knowledge. Great episode.
Yesterday I was half-joking about how an almost single-chip workstation could be built around the AMD MI-300 (24 x86 cores, GPU, and 128GB of HBM) and I think we could have some space for products that deliberately misuse these components.
Heck... Every hardware company should have a "tech misuse" department... IBM should do a Z-based desktop and a 2-core POWER chip, for instance.