What is the advantage you get from having a sophisticated software stack on a top of rack switch? I was under the impression that you only really configured vlans and maybe a span port every now and again. Someone care to enlighten me?
If only you could do this on EC2. >_<
For example, lets say you're moving VM's around, or needed more bandwidth between two areas. You could reconfigure the switch all in software much faster by doing it programmatically rather than going through even a standard CLI interface.
Want to design a new L2/3 routing protocol for your specific traffic needs? Now you can.
Think of this as the difference between a graphing calculator and a copy of R, or SciPy. Sure, they both can do the same basic calculations, but the latter has so much more programmability and headroom.
Some people may want to run OSPF or IS-IS, or do some kind of performance analysis, or do ARP/ND/DHCP proxying, or perhaps use switches to aggregate BMC monitoring & control, or who knows what.
Also, a software switch allows you potentially massively scale the number of ports on a single switch.