This is really, really cool and I didn't know the platform was open to the extent that you could install your own upstream Linux and just get going.

I am curious though what configuration option prevents this from ending up with software switching. I understand the mellanox kernel module was compiled and loaded, but certainly that doesn't mean that anything you do in the network stack gets converted to switch fabric code and uses hardware packet switching. How do you make sure that you don't errantly wind up with poor latency and capped throughput?

But also, short of making and selling your own networked device for whatever reason, what are the real benefits of going this approach? I can see crazy use cases for where you have full control of the network stack (but again, see my point above — how do you guarantee you are not doing this in software?) but for most purposes, especially with qsfp and fiber, how much are you really gaining by doing it on-device? What is the killer use case here?

EDIT

Upon rereading, it seems the switch is hard-coded to be hardware-switched and cannot end up in a situation where you are accidentally using software packet switching in the first place (i.e. it does not just optimize to a hardware packet switching state). But that limits what you can do considerably, to the point that an off-the-rack Juniper or CSCO or whatever probably has more features than you can do here without writing your own code to hook into the mellanox sdk?