> If they had been using New Relic correctly (note that I don't work for or speak for New Relic, I'm just a former employee), they'd have seen on the javascript-enabled monitoring that requests were taking a long time. The server-side time is only a portion of that, but it's clearly delineated.
Note that they were using New Relic as part of an expensive add-on package from Heroku. It gave them a queue time value in it's reports, but it was extremely misleading, since the only value it showed was for the router queue time, which should have always been extremely low (and was displayed as such). It didn't say "router queue time" or "dyno queue time shown elsewhere".
Since New Relic is supposed to be showing them everything that happens with their request on Heroku's servers, it seems logical that it would include dyno queue times.
Javascript-enabled monitoring would only show you that the request times are much longer than what Heroku says they are, then you still have to troubleshoot down to figuring out why.
> Unfortunately they picked a number (queue time) that doesn't have much functional basis on Heroku's stack at the moment, which weakens their argument.
I don't think it weakens their argument as it clearly shows that the biggest problem they have is not only out of their control (even with very short run-times, the higher the number of requests you have per minute, the more this problem is going to affect you) but that even buying very expensive tools integrated into Heroku's "stack" will not help you to see where the problem is. The tools were basically hiding the one problem that was solely Heroku's responsibility.
Do not forget that even while there were a lot of statistics about how long running requests can cause other, much shorter requests to take just as long and even timeout, the heart of the matter is that even with a high number of extremely short requests, the router can end up sending many requests to a single dyno while other dynos remain idle. There were plenty of graphs, even animated to show you the effect over time for this random dyno routing.