I am curious though what configuration option prevents this from ending up with software switching
The answer is "switchdev":
https://www.kernel.org/doc/html/latest/networking/switchdev....
The Linux switchdev driver is the awesome magic that says "make hardware-offloaded switching ASICs look just like software switching". It's beautiful and amazing, as you'd expect from Mellanox.