This paper largely focuses on percent load and mean service latency as metrics to optimize. But in web services you often care more about the tail-end latency, the p90, p99 etc. The math for this paper is a lot simpler because for mean service latency they only need to concern themselves with the mean service time for jobs, not the whole distribution. That seems like it would make some interesting follow up work to this paper.

Another thing is I think this paper is assuming the latency for joining idle queues is zero. This is another simplification that makes the math easier, because otherwise you have to worry about a processor sending a message to join an idle queue, but before that message gets sent the processor gets more work and is no longer idle. It would be interesting to see what impact that might have on performance.

I'm not too familiar with queuing theory, can anyone point me to work has been done on long tail distribution for this or other load balancing strategies?