alexforencich

These things are probably the best bang for your buck right now.

The ex-Alibaba KU3P boards are available for under $200, although you do need a separate JTAG cable with flying leads to load designs on them. I recommend starting with that. I'm building the next-generation version of Corundum on those boards, at least initially. I will certainly be adding support for Alveos and such in the near future. As far as adding functionality, it depends on what you want to do, but in general you should expect to have to do a fair amount of tinkering with the design.

This is correct, and the result of those streams has been released as corundum-proto here: https://github.com/fpganinja/taxi/tree/master/src/cndm_proto . Note that this simplified design is intended for educational purposes only, the "production" variants will be much more capable (corundum-micro, corundum-lite, and corundum-ng).

I'm using some very cheap (~$200) ex-Alibaba FPGA boards for the initial development of the next-generation version. They have 8 lanes of PCIe gen 3 and two SFP28 capable of operation at 25 Gbps. Honestly that's pretty hard to beat. Support for other boards will be added in the near future. The new library will also include a new IP stack called Zircon, which supports UDP, among other things.

The plan for corundum-ng is to support 400G, but it will be a while before this is available.

The GPU uses https://github.com/alexforencich/verilog-pcie + the Xilinx PCIe hard IP core. When using the device-independent DMA engine, that library supports both Xilinx and Intel FPGAs.

The GPU uses this: https://github.com/alexforencich/verilog-pcie . And there is an open-source 100G NIC here, including open source 10G/25G MACs: https://github.com/corundum/corundum

It uses https://github.com/alexforencich/verilog-pcie on top of the Xilinx PCIe hard IP core, which provides everything below the transaction layer.

A hammer, perhaps?

https://elixir.bootlin.com/linux/latest/source can be far more helpful than grep for this sort of thing

The graph above that one is totally wrong. The frame overhead graph may be correct, though.

Ah yeah, that's probably true. According to some back of the envelope math, it seems like the distribution should be more like 5%, 1%, 1%, 3%, 50%, 39%, ignoring the first and last size bins.

I think the author may have made a mistake in some of the math. The frame size distribution plots are likely based on the number of frames, not the amount of data contained in said frames. The 1500 byte and other large frames should therefore account for the lion's share of the actual data transferred. Correcting this error will totally change the final two graphs.

Well, we're planning on porting Corundum to the NetFPGA SUME hardware at some point in the near future. Should be relatively straightforward as the PCIe interface on the Virtex 7 is the same as on the Ultrascale parts.

NetFPGA does have a NIC reference design, but AFAIK it's just the Xilinx XDMA core connected to a Xilinx 10G MAC. No accessible transmit scheduler, no offloading of any kind, etc. Just about as spartan as you can get, and it's built from completely closed components so you can't really make many modifications to it.

For what we're doing, we can't use any existing commercial NICs or smart NICs because they can't provide the precision we need in terms of controlling transmit timing. We don't care about EBPF, P4, etc. We care about PTP synchronized packet transmission with microsecond precision.

Netronome cards are a gigantic PITA to program, and they are not capable of the precise timing that we need.

I don't know where the heck that price came from. The cards are more like $1200 new.

https://www.cdw.com/product/exablaze-exanic-x10-network-adap...

We have not done anything like that. Presumably that would be an important thing to do if a commercial product is produced at some point.

That's something we joke about quite a bit in the research group - "back to the future!"

Crosstalk is better than 30 dB, and double pass loss between ports is 5-8 dB. The switch is basically cycling through three or four different interconnection patterns that are defined by looped back fiber connections, so the signal has to pass through the switch twice.

Stradding is an artifact of very wide interfaces. On the Ultrascale+ parts, the PCIe gen 3 x16 interface comes out as a 512 bit wide interface. Every cycle of the 250 MHz PCIe user clock transfers 64 bytes of data. The issue has to do with how packets are moved over this type of interface. If your packets are all a multiple of 64 bytes, no problem, you get 100% throughput. However, if your packets are NOT a multiple of 64 bytes in length, you have a problem. What byte lane do packets start and end in? The simplest implementation is to always start packets in byte lane 0. The interface logic for this is the simplest - the packets always start in the same place, so the fields always end up in the same place. However, if your packet is 65 bytes long, the utilization is horrible - it doesn't fit in one cycle, so you have to add an extra cycle for every packet, and bus utilization falls to 50% as you have 63 empty byte lanes after every packet.

Straddling is an attempt to mitigate this issue. Instead of only staring packets in lane 0, the interface is adjusted to support starting packets in several places. Say, byte lanes 0 and 32. Or 0, 16, 32, and 48. Now, when you have a packet end in byte lane 0, you can start the next packet in the same clock cycle, but in byte lane 16 or 32. This increases the interface utilization. The trade-off is now the logic has to deal with parts of two packets in the same clock cycle, and it has to deal with multiple possible packet offsets.

The specific annoyance with PCIe packets is that the max payload size is usually 256 bytes, but every packet has a 12 or 16 byte TLP header attached, which really screws things up when combined with the small max payload size.

That's part of the research we're doing!

Take a look at: https://circuit-switching.sysnet.ucsd.edu/

And: https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENL...

The current generation of switches that we're working on uses diffraction gratings patterned onto glass hard drive platters, installed in a modified hard drive, spun by a custom motor controller that's synchronized to the NICs via PTP.

No free 40G MAC/PHY. Unfortunately, the Xilinx CMAC is 100G only, and the Xilinx soft 40G MAC/PHY is $$$$. I have looked in to building a 40G/100G switchable MAC/PHY, but it's going to be a serious pain in the rear.

Funny you mention that switch, we bought one of those off of eBay for our testbed as it supports PTP.

Also, for optical switching applications, one of the most important factors is how long it takes to bring up the link after switching. Because of this, we have no interest in spending time on 40G and 100G interfaces because interlace deskew takes hundreds of microseconds, and 100G also requires FEC which takes hundreds of microseconds to lock. So we're focused on 10G and 25G and running multiple links in parallel, which also provides more architectural flexibility. I added 100G support for three main reasons: the CMAC license is free, so why not?; supporting 100G makes the project a whole lot more interesting than only 10G or 25G, and it provides a simple way of testing the core NIC datapath.

There are two hard things in computer science: cache invalidation, naming things, and off-by- one errors.

Yes, I am aware of those. However, the kintex PCIe interface is a bit of a pain as it has a TLP straddling mode that can't be disabled, so it will be some time before it's supported as it will require some significant reworking in the PCIe interface modules. I am planning on supporting straddling eventually as this will improve PCIe link utilization on the ultrascale and ultrascale plus parts. If someone wants to donate a board, I can look in to supporting it.

It depends on the programming interface. JTAG is bit serial and rather slow, so it can take quite a while to load a large FPGA via JTAG. However, there are several other interfaces that can be used, including QSPI, dual QSPI, parallel flash, and a simple parallel interface from some other controller. These can run at many MHz and can load a configuration into a large FPGA in less than a second.

It's still in development at the moment. We'll see about the interface. But there are no plans to implement any segmentation offloads or TOE in Corundum, that will be left up to the network stack. However, scatter/gather DMA support is planned so that software GSO will work. Right now, most of the low-level twiddling is done from a user space app that directly accesses device registers. For a research device, that's fine, but that would obviously have to be improved for a commercial product.

Mellanox NICs (and basically all commercial NICs) do not do what we want. Software is not precise enough, and is on the wrong side of the NIC hardware queues. The whole point of Corundum is to get control of the hardware transmit scheduler on the NIC itself.

The group web page is here: https://circuit-switching.sysnet.ucsd.edu/

Corundum was originally geared more towards optical circuit switching applications, but it's certainly not limited to that. Since it's open source, the transmit scheduler can be swapped out for all sorts of NIC and protocol related research.

Corundum should work on any ultrascale or ultrascale plus FPGA with the necessary interfaces. Probably also virtex 7. We're considering porting to Intel/Altera parts at some point as well...mainly the PCIe interface will need work, but the rest should be directly portable.

Original motivation is to support optical switching research for datacenter networking applications. The research group web page is here: https://circuit-switching.sysnet.ucsd.edu/ . It is also mentioned in these slides: https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENL.... However, the design is very generic and should be interesting to applications outside optical switching. The main point was to get control over the transmit scheduler, coupled with a very large number of hardware transmit queues. There are a number of experimental protocols and similar that could benefit from this vs. implementation in DPDK.

It is still in development; not sure if I would trust it yet for production workloads. We will not be producing hardware; the design runs on pretty much any board that has the correct interfaces, including many FPGA dev boards and commercially available FPGA-based NICs such as the Exablaze X10 and X25.