As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Cedar".
As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Cedar".
I can understand how this developed. Things worked well for most customers. Many of those with problems got them under control with more dynos or multi-worker setups. Heroku's Rails roots biased them towards a "keep it simple, throw hardware at it, or look for optimizations in the app/sql/db" mindset. Well, many of their Rails/Bamboo customers complaining about latency, even in the presence of this growing issue, may have also (or even primarily) had other app issues too. (When supporting developers, especially many beginning/free-plan developers, it doesn't take long for your conditional probability P((we have a real problem)|(customer thinks we have a problem)) to go very low, and P((customer app has a problem)|(customer thinks we have a problem)) to go very high.)
Even when Heroku had a unitary (and thus 'smart') router, they surely got latency complaints that were completely due to customer app issues or under-provisioning, so they stuck with the 'optimize app or throw dynos at it' recommendation for too long. And, when they habitually threw more hardware at the Bamboo routing mesh, they were unwittingly making the pile-up issues for Bamboo web dynos worse. Some key data about the uneven pre-accept queueing at dynos was missing, which combined with habits of thought that had worked so far gave them a blind spot.
Despite the growing issue, adding dynos at the margin would still always help (at least a little) — as well as adding to Heroku revenues. Even without any nefarious intent, a 'problem' that fits neatly into your self-conception ("we give people the dyno knob to handle any scaling issues and it works"), and is also correlated with rising business, may not be recognized promptly. That's just a natural human biased-perception issue, not incompetence or dishonesty.
Break out Mathematica, Matlab or R and model the damn problem. Then go research the solutions already available (Hint: look at many grocery stores, queuing problems).
Much more useful than an apology is an acceptance of fault (which is not the same thing); an expression of desire to improve, and a sincere and demonstrable commitment to doing so.
[NB: don't mean to imply that Heroku have necessarily achieved all of that here]
-------
They apologized in another earlier post. https://blog.heroku.com/archives/2013/2/15/bamboo_routing_pe...
"We failed to explain how our product works. We failed to help our customers scale. We failed our community at large. I want to personally apologize, and commit to resolving this issue"
Exactly. If they didn't know, they should have. If they did know, well, ...