I'm very interested in how infiniband works out for you. I've considered it myself, as used infiniband switches are available for the same price as used 1g ethernet... but while I'm pretty comfortable with fixing ethernet when it breaks... I'd be much less confident in my ability to fix infiniband by swapping out hardware than I would be in my ability to fix gigabit ethernet. As far as I can tell, infiniband is something of a dying technology vs. ethernet, market wise.

On the other hand, infiniband is fucking awesome. DMA over the network, anyone?