Are you sure about that? They would on average perhaps be a slight bit faster, but if the randomized delays are an order of magnitude larger than the latency, they may not be able to do anything useful with that advantage.
If the delays are randomized then you have as good as chance of getting them as your competitors. On delayed times you will have a bad chance of winning, but in comparison to others with the same delays your latency beats theirs, on non-delayed times your latency beats non-delayed competitors and really beats delayed competitors.
Thus in the end the best latency should win biggest on non-delayed trades, and on delayed trades lose least.