Making a Linux-managed network switch

Ignorance is bliss.

I worked with one of these ASICs, from Broadcom. Not 40, but something like 4x10+24x1Gbps + PCIe to CPU. The ASIC cost as much as you would expect (I don't know the actual number - a couple hundred bucks?). The software interface to it was very poorly documented, and was a library that supported all Broadcom switch ASICs, so was a few hundred megabytes .a file and was full of functions that would just return "not supported on this device" errors, which you wouldn't know until you tried it.

OpenWRT these days can also be installed on some switches eg the Zyxel GS1900 series (though support for things like PoE and 10Gb/SFP+ ports might be limited, I’m not sure where things stand there).

Ah yes the legendary sprite_tm, I've come across this while writing my own raspberry pi firmware to manage the switch over USB.

This is more efficient if most of the traffic will get switched, seperate adapters is more efficient if most of it needs to be routed.

It'll be a combination of:

1. Throughput - say you use usb adapters, in a lot of ways usb is a shared bus so you'll run into max bandwidth quickly. This is especially because data will have to go in and then out, all the way to the cpu

2. Latency - because you're using software to do switching, it'll add time to process each packet and send it back out tne right place. You've also go any other interface latency adding to it

3. Power usage - eacj adapter will have it's own full network phy and hardware, which will increase the power draw. Combined with all the extra processing above and now your power usage is even higher. This means you also loae out on hardware offloading and other performance enhancememts that generally reduce power usage because less of the system is involved in mocing packets around

4. Features (potentially) - this will depend a lot on the hardware you choose, some of those cheap gigabit usb adapters i've tried didn't work with vlans and other features properly. But if you say load up a bunch of nice pcie cards with 1 or more ports thst support everything (never had issues with pcie ones) then you can now actually get a lot of features that are otherwise difficult or impossible on simpler hardware (though at that point you're doing routing more tham switching, but thay flexibility is why you'd potentially do this).

For such a low speed and small number of ports using a hardware switch circuit is not necessary.

Nevertheless, if the designing and building effort is neglected, I assume that the total cost of the hardware might be under $100, which is less than a computer with multiple interfaces would cost.

Still, 1 Gb/s networks are rather obsolete. One could make a managed network switch that is bigger and faster by using only off-the-shelf components for slightly more than $200, e.g. a 6-port 2.5 Gb/s switch can be made with an Odroid H4+ having 2 ports, together with its add-on M.2 card with 4 extra Ethernet ports. Another variant is to use a small computer with an N100 CPU and 4 2.5 Gb/s ports, which can be bought in this price range from various Chinese companies. Similar small computers with 6 2.5 Gb/s ports are a little more expensive, perhaps slightly over $300.

As everyone else is saying, power usage should be less this way. Probably less latency to traverse the switch than a software bridge, too. Switching should continue to function if the host OS crashes, combined with a watchdog and recovery, you could have a more available system where maybe some things don't work for a brief interval, but much better than a software bridge (assuming the switch chip doesn't crash or get stuck, anyway).

It depends on what your goals are though. If you want to inspect all traffic passing through the switch, having 4 interfaces is clearly better. If your host based switch is also doing a lot of communication, 4 interfaces gives you 4gbps from the hkst rather than 1gbps. Etc.

Ex-enterprise quad 1G cards are $15 or less on ebay. I'm partial to silicom quad bypass 1g_ PEG4BPI-SD; the bypass feature can be fun, and they're cheaper cause they're weird (you can mostly configure them to be 'standard nics' once and then plug them into anything without much fuss, but getting there can be challenging. Early ones come with pci ids set to silicom as the vendor and subvendor which makes them harder to use; the -SD cards have intel vendor id and silicom subvendor, so the normal driver will attach.

4x10g ports would be more to manage, and you might not have enough throughput for software bridging, depending on the host system. And quad port 10g cards are harder to find. 2x10G is reasonably priced though, if you're patient.

Your CPU will be in the data path there. Switches do their packet switching on ASIC rather than CPU. So it depends how good your CPU is, and generally that's not an effective use of whatever compute power you've got.

OpenFlow was kind of the wrong solution to the problem. (The right solution is switchdev or SAI.)

AFAIK OpenFlow tried to be far too generic, resulting in it not actually mapping onto any actual hardware without lots of abstraction leakage.

While you could make an OpenFlow client that uses this switch chip as the data-plane, it might not be the best as most of the features you would use OpenFlow for would not be able to use the relatively limited hardware fast-path.

Marvell 88E6393X works in 'dumb/externally managed' mode without firmware. You can use it with Linux switchdev like the one in this article.

I don't think this support it, only regular STP and apparently MSTP

The RouterBoard RB2011 is stuck on an ancient release version of OpenWrt, 19.07, see here: https://openwrt.org/toh/mikrotik/rb2011

The problem seems to be somewhat related to the NAND, which is IIRC somehow different from the supported RouterBoards. Somebody proposed a new solution to this problem, but that hasn't landed yet: https://forum.openwrt.org/t/wiki-cleanup-for-mikrotik-rb2011... and other threads.

I have a stack of them here, they are great to work with for their intended purpose but they are not that great if you want to run custom software.

I also grabbed a RB2011 diagram because it's a simple diagram that explains it well I think, the RB1100AHx4 is a better example technically since it's using the same switch chip but it's more confusing because they use the two CPU ports together and claim it's a 2.5Gbps link while it's two 1.25Gbps link and they ignore the encoding overhead.

The reason why I build this from scratch is that it that the expense is reasonable and this should end up in the FOSDEM video recording boxes and it has to fix a few issues specific to that design like it needing to expose 4 network ports to the front panel of the case but also be connected to the internal SBC. There's simply not that much space in the case to put passthroughs in the case to a switch to not have an external loop cable for the SBC and with a dumb switch the system can't be monitored anymore. Since quite a few of these boxes are built this becomes a reasonable solution (if you ignore design time, but this is volunteer work)