You can only do so much with gigabit ethernet. Add to that the fact that whatever apps you're running are competing for that same bandwidth and it's super hard to get good I/O out of it.
I keep hoping that somebody will do that, hook the other end to an SSD target, and sell me a 'high I/O' node. I want to stop dealing with hardware and colo facility contracts and I doubt I'm alone.