Coding Cloud Heroku May Face Lawsuit Over Bait-and-Switch Claims
wired.com
wired.com
I don't run a company, and I'm not responsible for maximizing return on my shareholders' investment. So maybe I lack the mental schemas that make these sorts of statements seem reasonable, but they always strike me as short-sighted. Yes, admitting wrongdoing may open you up to legal liability. But it seems to me that, in the long run, it's far better to take your licks and own up to your mistakes than to hide behind the shield of pending litigation. My gut instinct is to offer a sincere mea culpa and provide a transparent roadmap to improvement, and if you lose a court case then so be it. I'd like to think that the good will and respect you build from accountability and honesty will ultimately outweigh any short-term losses from court settlements or judgements.
As I said, I don't live in that world. I would be saddened, though not necessarily surprised, to learn that I was wrong.
The US legal system is setup that if a company admits fault in something like this, it makes defending yourself much more difficult. Meaning if a lawsuit is brought about, it's not about proving innocence anymore, it's about mitigating the damages you will have to pay. Your lawyer is going to want to have as many options available to him/her to deal with your case during it's lifespan, and admitting fault before the lawsuit even begins will limit the lawyers options when it comes to negotiations and what tactics they can use in the courtroom.
Given that each possible liability is its own case and stands on its own merits, it pretty much matches the simplest case of the prisoner's dilemma perfectly, IMHO... and so here we are. Sadly.
Might, in part, have to do with how lawyers get compensated. A short case or a quick settlement will result in very little compensation for them. Over the course of years and decades that matters so we see everything analyzed to death. At $500+ an hour. Per lawyer.
Don't like the legal method? Don't hire a lawyer. Try doing it yourself. See how far you get when the other guy's lawyers have done the exhaustive analysis and found the deciding element buried in a logical node on a distant subgraph you didn't even know was there to be explored.
It's a system set by lawyers for lawyers so of course you need a lawyer more than in many other countries. "Everyone" sues over everything
Thanks to the lawyers, no doubt. I am sure you didn't meant to say that if we pass some tort reform and limit some nonsense lawsuits (IP, malpractice etc) the country will lose it's wealth /power rank.
And with the rule of law comes lawyers. I know it's fashionable to hate lawyers, but they're a necessary profession.
Law is a field in which small details can have very large consequences once the full line of reasoning is unfolded. Because of the importance of exhaustively covering all lines of argument to the maximal possible depth, lawyers are required to do so.
They don't do it to bill you more. It is their legal duty.
If you don't like that lawyers are required on pain of loss of income, loss of profession and potentially loss of personal liberty to give you the fullest and most complete service that they are able to give you, then you are entitled to represent yourself.
For more examples, see: http://www.economist.com/node/21541423 - on attempts to reform tort law
http://www.economist.com/news/leaders/21571141-cheaper-legal... - on how US legal education can be reformed to reduce the cost of legal counsel
http://www.economist.com/node/18389167 - on problems with the US' patent laws
I was explaining why lawyers spend so much time minutely examining every tiny bit of a case for their clients. They have to. A lawyer who doesn't could find themselves in breach of fiduciary duty.
Also, read what I've said again. Nowhere have I said that the USA is rich and powerful because of lawyering. But neither has it stopped the USA from being so.
The reason lawyers tell people to clam up is because jurors put great weight on what people say about their own actions. It's precisely the other edge of the sword of what you're talking about--a positive public statement can have a great effect on customers, but one that comes out wrong can have a very bad effect on jurors when the plaintiff brings it up in court.
Just imagine being arrested. If you tell the police you are sorry- you are going to be found guilty. That is why all lawyers would tell you to remain silent until he arrives to the police station. The risk of admitting liability when talking is too big.
[0] https://blog.heroku.com/archives/2013/2/16/routing_performan...
Which is why fine print almost always makes even the nicest company sound like a jerk.
This is an admirable, well-intentioned perspective, but it is one that won't net you much benefit, not even in the long run.
Filing legal papers -- or, in this case, raising the banner for a class-action -- is serious business. By the time it happens, the time for defusing the situation and calling a truce is long past. The only opinion that matters is that of the judge and/or jury. And let's face it, being the loser in court is going to have far more impact on the public's opinion of the defendant than any honest-from-the-heart statement or outreach.
So with that said, there is nothing of substance that the defendant can say that will help them in court. The opposing attorney is not going to say, "Well, I was going to grill you, but I've had a change of heart ever since reading that impassioned, honest, salt-of-the-earth statement you released pre-trial". The judge won't care either. And the jurors aren't supposed to even be aware of it.
I think a civil trial is not much different than the situation portrayed in the famous "Don't talk to police" video:
http://www.youtube.com/watch?v=6wXkI4t7nuc
The people you're facing in court don't care that you've made an effort, post-lawsuit-filing, to be upfront and friendly. Anything you say will not help you, it can only hurt you. In this case, Heroku's reticence in speaking their mind outside of court is the right one.
A mea culpa right now might haunt you in the long term.
Maybe I am misremembering...
The whole thing seems strange to me. I don't really understand what Rap Genius hopes to gain here. If Heroku wasn't providing enough performance for their money (regardless of technical cause), then why did they stick around so long?
Would Rap Genius still have sued if the documentation had been 100% correct and instead the problem was just plain old slow I/O on Heroku's side?
We didn't know how bad the performance was because Heroku's tools (logs and New Relic) reported incorrect performance data (i.e., they said requests weren't queuing when they were actually spending a ton of time queuing)
EDIT:
> Would Rap Genius still have sued if the documentation had been 100% correct and instead the problem was just plain old slow I/O on Heroku's side?
To be clear, we have no official association with http://herokuclassaction.com/ or the lawyer behind it – I actually found out about the site from the article's author when he interviewed me for the story.
(But I do think Heroku owes its customers a refund)
Fraud is usually a criminal matter.
This is a civil matter. Probably under torts law, especially if there's anything in California/US torts law that resembles Australia's tort of "misleading and deceptive conduct".
Fraud and torts are different. And to forestall the next obvious point: torts and contracts are different.
> I don't really understand what Rap Genius hopes to gain here.
A refund.
Heroku's selling pitch is linear scalability. Rent 1 dyno, get ~ (1 dyno / Σ dynos) improvement in performance. Rap Genius demonstrated that they were getting exponentially worse increments of added performance per dyno added.
> If Heroku wasn't providing enough performance for their money (regardless of technical cause), then why did they stick around so long?
This is orthogonal to the question of whether Heroku acted in a legal way.
> Would Rap Genius still have sued if the documentation had been 100% correct and instead the problem was just plain old slow I/O on Heroku's side?
Courts don't worry their heads with strawman hypotheticals.
(IANAL, TINLA)
Rap Genius isn't the ones doing the suing. Rap Genius wrote about it, which caused Heroku to come clean. I think Rap Genius already got what they hoped to get (i.e., exposure, official comment, etc).
> "I don't see any obvious or intentional fraud."
You don't need intent to be sued - you only need to have caused damages to the plaintiffs. What Heroku has effectively done was:
1 - misrepresented their product in a substantial way.
2 - "overcharged" their customers substantially by giving them less performance than they paid for.
Both of these are material damages, and there's a more than reasonable case for Heroku's customers to recover this money.
This isn't a criminal case here, there doesn't even have to be fraud (as defined in the criminal law context).
As far as I understand that is not really the case. The newer version is the Cedar stack which supports concurrent apps where Bamboo only supports non-concurrent apps. This would reduce the problem with random routing, but not eliminate it.
Customers still have a problem, but Heroku doesn't from a legal / class action-y perspective.
Why is Heroku "in the wrong" exactly?
It looks pretty bad when your customers complain about problems that are hidden by your own monitoring tools, and then you tell them to buy more crap to fix it! It's not a small amount either -- as mentioned the queuing time was larger than the request time. And on top of it you're paying out the nose for this crappy monitoring tool.
EA is basically a lost cause; their business model and culture is the problem. Having better sacks at Dachau wouldn't have helped the Nazi image, either (to auto-Godwin)
I'm pretty surprised they didn't offer refunds already. It was pretty clear they were asked for, and with no update by now, it's safe to say they were refused. That seems unreasonable to me. If you were in the position RapGenius, would you honestly not have expected a refund already?
I dont use Heroku, but if you read Rap Genius' original paper discussing the queuing problem, they specifically stated that there is nothing that they can do about it.
A third-party could do a load balancer entirely outside of Heroku or even AWS, feeding sessions in based on feedback from the application ("intelligent routing"). Heroku even offers a way to bill for that kind of thing. You can open outgoing connections from each dyno and then load-balance over that.
Doing it with ELB would be possible I think, but I don't really like AWS ELB. I think it would make the most sense to do near US-East AWS, but arguably a CDN type service could do it from many locations. Being closer to users (as long as your connectivity to US-East/Heroku is still good) helps performance more.
Rap Genius says they aren't involved either, so this is probably a single-person law firm looking to create some buzz for himself. He should threaten to sue Amazon next for all the times they haven't held their end of the bargain. That'll get him a ton of publicity.
At the same time, it's an edge case in the economy. If you harm a large number of people a small amount, they won't sue you to recover for their injury.
From the article:
As for Lehman, he wants some of the $20,000 his company pays Heroku each month returned. “We deserve some sort of refund here,” he says.
EDIT: I re-read the article, and it's not even clear that Rap Genius is part of this. Looks like a lawyer trying to drum up support.
http://news.ycombinator.com/item?id=5237920
http://news.ycombinator.com/item?id=5241686
Really, though, given how much money Heroku charge and how much money Salesforce.com would be able to stump up in a settlement or judgement ... it was inevitable that a fast-moving law firm would jump on this.
For example, if we both enter into a contract, you can hold me to my end. If I fail to hold up my end without intending to fail, then you can still insist that I make right or seek remedies for my failure.
(IANAL, TINLA).
Ridiculous? Well consider that it gives a strong incentive to ensure that claims made to customers are somewhat accurate. Reducing the cost of caveat emptor makes society richer by reducing transaction costs and so encouraging trade. Sellers have a strong informational advantage over buyers; in general the cost to them of making a correct statement is much lower than having each customer having to find out the truth independently.
Not that this is the purpose of torts law, which evolved mostly to fill glaring, unjust gaps in the remedies of the older forms of common law. But that is the social and economic function that it has come to serve.
(IANAL, TINLA).
But the founders of Heroku have no more liability than the company in control of its assets, which in almost all cases would be some sort of Limited Liability vehicle.
HIEROKU!?
Ignoring the funny url. I doubt this will go anywhere besides in the domain expiry list.
As unlikely as it is, I hope that Heroku terminates the account of anyone that joins the class action against them.
Really? Because it really disgusts me when a company:
* Misrepresents their product (saying it's A when it's actually B)
* Privately acknowledges this misrepresentation under the radar
* Makes large and ridiculous amounts of bank on this misrepresentation (since you have to throw more dynos at the app to mitigate the lag caused by the misrouting.. and Heroku may be many things, but economical it isn't)
* And on that note, provides very expensive monitoring and performance analysis tools that further perpetuate the misrepresentation!
* And then force you to resort to litigation to get some portion of this huge mis-spent investment refunded. (Looking at the suit, this is all that's being asked for)
This wasn't a mistake on Heroku's part, this was calculated. And even if you accept that it was a mistake, the fact of the matter is that their negligence in this matter made them a hell of a lot of money.This is definitely grounds for a suit.
This lawsuit will go nowhere and it is sad that such a supposedly tech savvy crowd wants to bring out the pitchforks so fast. But I guess since Heroku has a lot of cash they are now the enemy since they've made it and they should be taken down.
The further problem it looks as if they knew the claims were untrue.
These are not matters of internal consideration. They cut directly to the heart of the Heroku marketing material: "Write for our platform, our expensive secret sauce will make your stuff linearly scalable."
In which case people expect that it will be linearly scalable. Not exponentially anti-scalable.
If my instances don't launch "quickly" or I've designed my app in such a way that it isn't 100% linery scalable I can now sue Amazon? You can't sue heroku for their marketing speak.
You can sue anyone for anything, success is not assured.
That bit of pedantry aside, Heroku led people on to believe that their queueing backend was intelligently distributing requests to nodes. Rather than randomly.
You are not allowed to argue this, as Heroku themselves acknowledged it!
Furthermore, they blatantly lied about it. From the rap genius article:
Heroku claims that though they received reports of unexplained latency over the past couple of years, they weren’t able to figure out the request queuing issue until they read the Rap Genius article. But there is evidence that Heroku knew about the problem for more than 2 years.
And the conclusion, also from the same article:
Based on Heroku's response so far, it appears their approach to fixing the problem is, “We promised you intelligent routing, we delivered random routing (which is worse than intelligent routing), so we're going to change our documentation to make it clear that we're only promising you random routing.”
This works for future customers, since once Heroku makes these documentation changes, everyone who signs up will understand exactly how routing works. But it does nothing to address the time and money that existing customers have spent over the past few years. What does Heroku owe them?
Heroku themselves!
Again, they changed their documentation.
Marketing speak is not allowed to be misleading or flatly untrue (as it was in Heroku's case). This can and has been the basis of successful lawsuits.
I don't see much of anything in a lawsuit unless specific metrics related to SLA's that were agreed to were broken.
That's generally considered uncool.
Uh.. no.
Heroku claimed to use intelligent routing, they used random routing. And then covered it up with their monitoring tools.
How in the name of crap can you blame this on the user?! Did you even read the Rap Genius writeup on this problem? If I were in their position, I would be utterly livid.
Define "intelligent routing". Maybe it is intelligent to route 85% of your requests to /dev/null. Can you sue because your performance sucks in your application use case? Most normal people eval their provider, try to get answers from support (which rapgenius have done here), and if the answers aren't up to snuff or the performance isn't whay they think it should be for what they are paying they move to another provider. They don't sue their provider for being "slow".
At least Heroku themselves define "intelligent routing" such that the request will only be given to the free Dynos. Their random routing does not do that.
It would be hard to give worse legal advice than this.
Now that's bad legal advice.
"Bait and switch"? If a couple of Dynos is not enough for your load, you should be moving to self-managed hosts
I have to imagine they're involved based on the number of articles I've read about them over the last week. Their own article, tech blogs interviewing them about the article, and now the founder's picture at the top of this article.
He’s spoken with one customer, who has since backed off. “That being said, I am in the process of drafting a complaint, in case their feet warm up,” he told us.
Furthermore, Tom (the dude pictured) commented in a thread above saying he didn't even hear about the lawsuit until Wired called him to interview him for the article. Consider putting away your "Jump to Conclusions" mat and joining us in the world of facts.
And they're not even involved with this lawsuit.