How We Got Owned by a Few Teenagers (and Why It Will Never Happen Again) | Hacker News Reader

How We Got Owned by a Few Teenagers (and Why It Will Never Happen Again)

blog.phpfog.com

405 pointscardmagic189 comments

I feel really bad for the phpfog guys. But given the situation, I think they handled it admirably well - kudos to them. No software is secure and this could have happened to anyone. Especially startups who have to take shortcuts at the very beginning.

I know the attackers were just kids but I have to admit pursuing legal action sounds very tempting - even to just act as a deterrent to others. If they had just put up phpfogsucks.com, it might have been ok. But tweeting trash from their twitter account, redirecting their root domain to phpfogsucks, etc - are all not cool at all and should have some consequences.

What I find most disturbing about this whole situation is the way in which these teenagers are handling themselves, especially after the fact. The continued denial of responsibility and half-hearted mea culpa, coupled with the monetary damage to those businesses who had been running on PHPFog, leads me to sincerely desire that these teenagers face a penalty of some magnitude, not just a slap on the wrist.

Maybe then they'll stop with the half-assed apologies and recognize that there's a right way and a wrong way to do things.

Maybe then they'll stop with the half-assed apologies and recognize that there's a right way and a wrong way to do things.

PHPFog built a castle out of sand and you're upset that a wave came and demolished it. I'm always surprised at how thin-skinned a lot of HN commentary is. "Oh, Zed shouldn't be so rude" "These kids' lives should be destroyed for playing games with an wholly insecure website." "I stopped reading that article because it used the word blowjob."

I don't get angry at my dog when he shits in the house. Being angry at something that can't understand only satisfies the urge to shift blame.

My dog shits in the house and it's my fault for not walking him sooner. If some children compromise every level of your company then getting mad at them is only trying to deflect the blame. PHPFog is the only responsible party in this mess. I feel for the customers who still trust them.

This isn't a wave knocking over a sandcastle or a dog shitting in the house. These are 16 year old kids, old enough to know right from wrong, and with the knowledge and skills to exploit the system. And once the exploit worked, they didn't then responsibly disclose the problem to PHPFog; they started vandalizing, changing passwords, and the works.

This is like someone finding an unlocked door to the apartment building's maintenance office, taking the master keys from there, rifling through a bunch of people's personal belongings, sticking signs in the windows saying "this building's landlords suck," and changing the locks on some of the doors to make it hard to clean up the whole mess.

They absolutely are the responsible party; you should never blame the victim of a crime just because the victim didn't take adequate steps to defend themselves. If I accidentally leave my door unlocked one day, that does not make it suddenly OK to come in and take my stuff and it's my fault for not having locked my door, instead of yours for taking my stuff.

Now, in this case PHPFog does bear some responsibility, because they have a duty to protect their customers as well as possible, and from reading about how this happened, it sounds like they were amazingly sloppy and irresponsible about it (passwords stored in the clear on the server, passwords shared between various accounts, leaving unsecured shared systems running after beta launch, etc). But that doesn't reduce the culpability of the attackers; they acted maliciously, with full knowledge of what they were doing, vandalized systems, changed passwords, and bragged about it.

I'm now nearly 25, and the amount I have changed since I was 16 borders on the immeasurable. Teenagers are glorified children. We seem to forget how little reflective capacity we all had when we were teenagers.

If I were 16 and hacking some stupid website, I would think, sure this is "wrong", and I might get in trouble, but I probably wouldn't think that it would matter 5 years from then, or 20. Truth is, a criminal infraction can fuck your life up worse than is deserved. You could be forbidden to immigrate to your wife's home country, denied any worthwhile job you can find, or spend a sizable portion of your life in a smaller cage than they house animals in.

There's a reason juvenile law exists, but we forget it. We charge 12 year olds as adults. This isn't that different.

Hell, where I live, a lot of employers will check the sheriff's website for public arrest records. These aren't convictions, they're arrests. People are being denied jobs based on accusations.

I've never gotten so much as a speeding ticket, but even I realize what a load of shit this all is.

So when they turn 21, a "be responsible" switch will magically flip in their minds?

The job of parents and society is to teach children responsibility. That means having consequences for your actions. And the closer you get to adulthood, the more adult those consequences should get.

When I was a teen, some real estate developers tore down a bunch of woods where I had always played and started building a house. I was ticked and I vandalized the construction site. But my crime was discovered, and I had to work carrying lumber in the hot sun and scraping glue off windows to pay back the damage. Why did I have to do this? Because the developer talked to my parents. And my parents made me do it.

This was light punishment - I wasn't taken to court, and I didn't get a criminal record. But parents are to children as legal system is to adults: they set the rules and enact the punishments. If they don't, someday the legal system will have to address their failure to do so by locking up their grown children who never learned right from wrong.

I'm not sure what the consequences here should be, but the argument "they're kids, they can't be held responsible" is silly. If they're not expected to be responsible, they won't learn to be responsible.

What you're saying is fine. Its probably the better adult way to handle it - talk to the parents. Other people are suggesting FBI/Criminal law. How would you have fared/grown up if they sent you to court?

Do note that since he is tech oriented, he probably now knows what a S* storm he has kicked up - the blog response pretty much ensures that he is aware. I assume you would be sweating bricks if you knew that the FBI was coming after you AND that you had hand delivered a bullet point confession. I assume thats a pretty strong deterrent (in his specific case)

(Incidentally fighting to stick up for your woods was a kinda nice thing to do, modulo the amount of vandalism you pulled off.)

I see your point, but not all teenages would do this even if they thought they would not get caught.

While I agree that the full force of adult law is probably inappropriate, some punishment should be exacted. Perhaps some sort of injunction regarding limited access to the Internet for a time (to be enforced by their parents, and to include replaying any fancy 'net capable phones with a good old not-even-got-GRPS mini-brick phone for the duration).

If I left my front door open and some kids came in, raided the fridge, broke the TV and ran up a huge phone bill, I would not expect them to get away completely unpunished. I'd not want them put away, and I wouldn't want it on their permanent record unless something particularly immoral was done (harming the cat, for instance), but I would want something to be seen to be done to educate them on right/wrong and act as a deterrent to others. I would also expect to be laughed at for being daft enough to leave my front door open!

  These are 16 year old kids, old enough to know right from
  wrong [..]

These kids were being assholes, but that does not warrant federal charges. The problem is that we can choose between unsatisfactory public shaming or thoroughly ruining their lives. There is no middle way, where they get an appropriate punishment, fitting the damage done. If harsh punishments were a deterrent, these kids would already have been deterred, because there are plenty of examples of teenagers harshly punished for relatively minor computerrelated crimes. I'd rather see them grow up to become, probably average, members of society.

Also, whether they know right from wrong is a question whose answer definitely isn't as clear-cut as you make it out to be. There's a reason we don't consider them adults yet.

  This is like someone finding an unlocked door to
  the apartment building's maintenance office, 
  taking the master keys from there, [...]

Yes, in theory, it is. But in practice, it's not. Especially if you're still a kid. It's easy to miss that there's something going on in the real world when you're doing damage "just online". (I know I sound like the "You wouldn't steal a TV! Why do you download movies?" crap-ad, but I hope you get my point.)

I've been a 16 year old idiot myself. After doing some stupid things to a website I was threatened with some trouble. I could avoid it by having extensive talks with the site's owner, and paying for their losses, but still, I learned my lesson: everything you do on the internet, in the virtual world, has an effect in the real word. Also it was an urgently needed wake-up call for me; I learned to think twice and since then haven't done any nasty stuff without considering its impact.

So in my opinion some sort of punishment is needed for the kids to learn their lesson. The FBI or other federal institutions shouldn't be involved. Talk to them, I'll guess they're nervous as shit right now! Just bill them the time you needed to tidy up the mess they left (or a fraction thereof), and I think both sides are good.

I really like the unlocked door analogy, but there seems to be some kind of disconnect in everyone's mind when it comes to "online" crimes.

Door locks are extremely "exploitable", but if a 16 year old were to use a bump key to gain access to PHPFog's corporate office and vandalize the place, all of the sudden it's a much bigger deal.

I have a theory that it has to do with familiarity and empathy. Door locks are a pretty standard solution. We all have them on our homes, and we think to ourselves, "I've done a reasonable job of securing my home." When someone's home/office is broken in to, we can easily identify with them. We look at the scenario and realize that we could easily suffer the same. We empathize with them.

Move the playing field to the Internet and all of the sudden everyone is expected to have Fort Knox level security. When someone's infrastructure is compromised, everyone stands atop the high hill, looking down on the drowning masses as the tide comes in, but the reality is that we're all vulnerable at some point.

A startup could easily spend as much on security as they do developing their core product. Why? As a startup, I'm not going to invest in double-reinforced steel doors, bullet proof glass windows, armed guards, and a centralized vault. That's wasted money in my view, because I have a reasonable expectation that people will act with civility. If someone does break in, I'm insured, and I will report the crime to authorities who will investigate. If the criminal is caught, there are real penalties, and they'll carry the stigma of having to check "YES" next to the "Have you ever been convicted of a felony" on their job applications.

I'm not saying we should try these kids as adults, but when I was 14, some kids who shared a bus stop with me broke in to a house near our bus stop and trashed the place. They got caught and suffered some severe penalties. It was a valuable lesson for everyone involved. A couple of the kids were from really bad homes and suffered from greater influences than the threat of the law, but the other two turned their act around really quickly. Had they gotten away with it, or had the attitude been "they're just kids", I'm not sure they would have realized the impact of the crimes they committed. I think we need more of this balance in our views of internet crimes.

If the landlord leaves the master keys unprotected, he most definitely shares some of the blame when the apartments of his building are broken into.

While there's a lot of emotional appeal to seeking justice in this case, my inner pragmatists says that this should really be looked at as free penetration testing for PhPFog. If this kind of hacking had stiff penalties (as you desire), only those with truly malicious intent (and probably financial motive) would do it. Likewise, the consequences wouldn't be some petty vandalism, but serious financial damage.

The fact remains that the site was insecure enough for a 16-year-old to find his way in. And the contributing factors to this insecurity might not have been identified had he not performed the attack in the first place.

"you should never blame the victim of a crime just because the victim didn't take adequate steps to defend themselves."

This reasoning is not applicable when the victim is a corporation who gives implicit or explicit guarantees to their customers about security. Your example should be: if you stored your stuff at a paid storage facility and someone there accidentally left the door yo your unit open.

I wouldn't care if it's a bunch of teenagers or Chinese cyber-warfare team who breaks into my Gmail account, I'd be mad as hell with Google for letting this happen.

Pressing charges against the proprietors of the crime will not change the fact that this host acted in a very irresponsible and sloppy way, and is in short untrustworthy.

I work with 16 year olds on a daily basis. Many, if not most, of them have a severely underdeveloped sense of morality, and know "right" from "wrong" no more than adults dictate to them.

"PHPFog built a castle out of sand and you're upset that a wave came and demolished it."

Your analogy is slightly off. A wave is an act of nature: this is more along the lines of a jealous kid who knocks down someone else's sandcastle because he can't build his own.

"I don't get angry at my dog when he shits in the house. Being angry at something that can't understand only satisfies the urge to shift blame."

While some HN posters might feel the the 16 year olds involved in this incident have the same mental capacity as your dog, I'd like to give them slightly more credit. ;)

I can only speak for myself here, but I do think the people involved in compromising PHP Fog should be punished. No, I don't think they should get life in prison (</hyperbole>): I hope they can learn from their mistakes. However, they did commit a crime, as they've admitted both here and elsewhere online. They should be capable of understanding that their actions have consequences, so I think some consequences are in order. What those consequences should be is up to PHP Fog.

Your analogy is slightly off. A wave is an act of nature: this is more along the lines of a jealous kid who knocks down someone else's sandcastle because he can't build his own.

In terms of moral culpability, sure. But when I put systems on the internet, I basically treat "intrusion attempts" as in practice part of the environment, like "mosquito bites" are in Texas. Perhaps they're best thought of as kids knocking down sandcastles rather than ocean waves, but their ubiquity makes them feel more like ocean waves, because you can basically assume that there are tons of those kids, and they're going to kick at your sandcastle every day.

The fact that there's a whole ecosystem of bots running automated intrusion attempts makes them feel a little bit force-of-nature-ish as well. If you lived in some neighborhood where thousands of roving robots were constantly checking doors to see if they could find an unlocked one, you'd have to treat "roving robots" as a quasi force of nature. Well, either that, or come up with a policing method that finds the controller of the robots and shuts them down, but I have relatively low hopes for how much of a dent "cybercrime" policing will make in the overall online-intrusion ecosystem.

From the perspective of security protection, intrusions are an act of nature. You should be no more surprised at an especially strong wave than you are at an exceptionally immature child.

I don't think we're using the same definition of "act of nature":

http://en.wikipedia.org/wiki/Act_of_God

"Act of God is a legal term for events outside of human control, such as sudden floods or other natural disasters, for which no one can be held responsible"

Do you think nobody can be held responsible for this breach?

You're right, we're not using the same definition. That doesn't mean I don't have a point.

Of course. I guess I'm not clear what your point is though.

My point is that you must treat intrusions as an inevitability when trying to counteract intrusion. And anyone who builds a sandcastle should be aware of the ocean. The kid's breaking into this account is embarrassing.

Just because we can hold individual humans accountable (and should) doesn't mean we shouldn't have the perspective of "CONSTANT VIGILANCE."

Still, does that really hold up?

Certainly should have to treat intrusions as inevitable in designing the system, but there still is responsibility on the part of the intruder.

I lock my door because I consider it inevitable that someone will eventually try and break in. However, if someone does break into and vandalize my apartment, I sure as hell would consider them responsible and not consider in an act of God.

You seem to be implying that since PHPFog should have defending against this, that what the teenagers did is perfectly acceptable.

I never claimed it was acceptable. Only that it was irrelevant. Why should anyone besides PHPFog's lawyer and the kids' parents care? It's because PHPFog chose to play PR guru and throw the drama into their postmortem as a distraction.

Does it matter to you if some kid in Australia is brought up on charges? No?

Does it matter to you if a hosting company is competent in securing their servers? Yes?

Any discussion of who did the hack servers no purpose other than to distract from the only issue that matters to anyone which is PHPFog's security.

Well, legally no - I think 16 year olds _aren't_ "held responsible" for pulling crap like this (rightly or wrongly).

PHPFog fucked up security, that's a given. And they should be thinking carefully on the lessons learned from that.

But your post seems to imply that the kids who did this have no responsibility for their actions; "it was just waiting to happen". This I strongly disagree with (not least because I work with 16 year old kids and they are completely able to take some level of responsibility for their actions).

Sure, at 16 your world view is incomplete and you can make rash decisions ("for the lulz") that backfire bigger and faster than you imagine. On the other hand there is no doubt they knew the illegality and the ethical issues with undertaking this - even if only vaguely.

And if they do not completely understand those issues, do you not think they should be taught them? As responsible adults we should be getting across to them in a sensible fashion that this was not a nice thing to do, and that the impact could have been a lot wider than it was.

Because if we don't and next time they do some real damage, well, that was an opportunity lost.

FWIW I think he crossed the line by causing damage. If I caught a kid breaking into my house I would probably drag him home to face his parents. But if he started smashing plates I'd be a little more pissed, that is a wanton act and probably needs a more severe punishment.

Sure, criminal charges are a silly approach in this case (no need to ruin his life for one silly mistake). Call his parents, explain what has happened and then get him to do some sort of "community service". That's an important lesson in consequence.

In this modern world 16 year olds are not nicking alcohol from the corner shop any more; they are breaking into websites. And that has potential for much more dramatic and widespread impact. We need to stop saying "oh, they're only kids". Instead we should recognise that 16 year old "hackers" exist on the internet and think of ways to communicate with them (ideally in a way that gives them an avenue for their curiosity without risking too much damage :)).

Bottom line; a 16 year old kid is a far cry from your dog when it comes to the ability to "understand" what you have done.

Just my 2p :)

I'm sorry, but in between your Straw Man argument and your indirect Ad Hominem attack, I fail to see you address the point that these kids caused harm to a business. Nowhere did I say that PHPFog bears no responsibility for the security of their service, but that doesn't excuse what these kids did one bit. I'm just much more impressed with the way that PHPFog is handling their business after the fact than these kids are. Nowhere did I suggest that these kids lives should be destroyed, however I do believe they most definitely need to be held accountable for their actions. PHPFog will be held accountable for their actions by whether or not businesses decide to do business with them going forward.

Both parties bear responsibility, and it's absurd to think otherwise.

>I fail to see you address the point that these kids caused harm to a business

Very well. These kids caused harm to a business. So what's that change? The business screwed up, badly. The agent of destruction is quite irrelevant. Had it been a power failure, backup failure, permissions failure, data leak, or data corruption would PHPFog deserve any less blame? This need to shift some responsibility to a bunch of kids is nauseating.

>I'm just much more impressed with the way that PHPFog is handling their business after the fact than these kids are.

This is another example of the weird HN mentality when it comes to companies "apologizing" (Like WakeMate blaming their Chinese manufacturer for flunky power supplies). Are you actually impressed that a corporation has better PR than a bunch of children? Does that even make sense to you? I'd be impressed if they had managed to actually apologize while accepting all the blame without trying to pawn off the responsibility for their mistakes on some kids.

> This need to shift some responsibility to a bunch of kids is nauseating.

They aren't shifting responsibility. The kids are responsible for their own actions. They did something illegal. They are responsible for it.

Now, PHPFog is also responsible for protecting their customers; they are supposed to provide a secure hosting environment. PHPFog is a victim here, but has also acted irresponsibly with regards to security (not criminally irresponsibly, but if harm did come to their customers due to this, there could be possible civil liability). The fact that PHPFog bears some blame for their security practices doesn't take responsibility off the kids who broke in and vandalized their systems.

With security that lax at PHPFog, it was inevitable that someone would have broken in. In that sense PHPFog was lucky... had these security problems not come up now, and be exploited by little punks with no larger agenda than vandalism, there could have been much more more serious damage later. What if some cyber criminal gang had turned their attention to PHPFog, and been a bit more subtle about the breakin?

I was appalled at the frequent mentions of 'luck' in that blog post. Your job as a sysadmin is to eliminate luck. To eliminate chance. To make _sure_ everything stays running, everything stays secure, everything stays confidential.

>They aren't shifting responsibility

Yes PHPFog is. The only reason these kids are even mentioned in the blog post is to shift blame. Their part in the post serves no other purpose.

The entire event could have been recounted without a single personification of the hackers in the blog post.

No, they are mentioned because they are the criminals who intruded and vandalized the system! Without them, none of this would have happened. There's no shifting of responsibility, since the kids who vandalized their system are responsible for their vandalism.

I can't honestly imagine what kind of moral system you have in which you don't believe that criminals are responsible for their own actions. If someone breaks into your home, is it your locksmith's fault, or the police's fault, or your alarm company's fault? No, it's the fault of the person who broke into your home. Perhaps one of the other parties mentioned was negligent, or perhaps not negligent but they could improve their security practices (install stronger locks, upgrade your alarm system, do more patrols in your neighborhood), but it's still the fault of the person who broke in and vandalized your home.

>No, they are mentioned because they are the criminals who intruded and vandalized the system!

"They" are not relevant in any way! "They" are tabloid meat for an internet drama. "They" are a distraction from the fact that a hosting company had piss poor security surrounding the core product. The entire story could be told without mentioning the hackers by name, or providing any biographical information. The only reason to include them is to distract from the real issue.

Replace "16 year old" with Russian, Chinese etc. Yes, vandals are bad. That's not exactly in question. In question is the sheer gall of PHPFog to shift blame to some kids to try and cover their embarrassment.

I think the blog post should be rewritten. Instead any mentions of the hackers should be completely neutral. Then it will be PHPFog getting out there and taking responsibility for their mistakes.

No distraction. No hand waving. And no tabloid drama.

Agree - creating a tangible identity to the villain is pulling heat off of PHPFrog.

Lets put it this way - a 16 year old kid who got lucky broke their site.

If it was anyone with intent, we would not know.

This is a case study in how to handle a situation like this. Its brilliantly done, inclusive of the comment where he says "the community is standing by us".

Its actually a brilliant brilliant PR piece.

I couldn't have said it better myself.

Perhaps a little mention would be good, but the way they make it the key points of the post, and so many people commenting on it accept it, disgusts me.

> PHPFog built a castle out of sand and you're upset that a wave came and demolished it.

Technology analogies invariably suck, but I'm pretty sure this is provably better:

When a teenager smashes in a storefront window, do we say they should've had bars over it?

The major problem with your analogy is that storefront windows don't have hundred or thousands of bricks thrown at them everyday. Web hosts are basically under constant attack. Would you suggest that the CIA, NSA, etc. not worry too much about their computer security? If not, then I don't see why you would imply a web host shouldn't be expected to secure their servers as much as possible either.

If a storefront was under constant attack, then yes, they should have bars over it. In fact, in my hometown, there was a streak of vandalisms where kids were throwing bricks through windows. After getting hit 3 times, one store replaced their huge glass windows with smaller plexiglass ones.

I think you're missing the point of the analogy. Check out http://en.wikipedia.org/w/index.php?title=Victim_blaming&...

I'm well aware of victim blaming, but sometimes it's justified. If someone hacked into your bank account and stole your money because of a security flaw your bank decided to put off until later, surely you would lay blame on your bank as well, no?

Let me be perfectly clear, lots of the blame lies with that attacker. But it is also the responsibility of the a web host to fortify their systems sufficiently, which clearly wasn't done in this case.

Let's not forget that the victim here is PHPFog's (potential) customers and not PHPFog itself.

I can agree with you that PHP Fog's customers were affected and are thus victims. I don't understand how PHP Fog isn't a victim here though.

I promised myself I'd avoid another analogy but...

If I give the bank my money and the next day I get an email saying "Sorry, we didn't feel like locking up last night and some kids looted the vault." I'd have a hard time calling the bank the victim.

And something concrete: PHPFog knew the holes existed and were negligent. I would say they even have some contributory negligence (IANAL). Especially after admitting they knew they were vulnerable.

You know what we call kids who loot bank vaults? Bank robbers.

No, we call them juvenile delinquents and treat them like children.

And we certainly don't call the bank the victim.

1. We don't treat them like adult criminals (necessarily), but we certainly treat them like criminals. Here are some examples (some harsher than others):

Bank robber, 13, could get 21 years in US jail (http://www.breitbart.com/article.php?id=CNG.cb17379375828ffc...)

Teen bank robber to be held for two years (http://www.morningjournal.com/articles/2011/02/18/news/doc4d...)

Boy, 15, Charged in Armed Bank Robbery in Lancaster (http://articles.latimes.com/2002/aug/09/local/me-bankrob9)

2. Why do you think banks aren't considered victims in cases of robbery? Because they could have done more to prevent the robbery from happening?

You're conflating armed robbery, a violent crime, with hacking a web site? That's the end of this thread.

Dude, you started the metaphor with tour comment. Can't exactly blame someone for calling you out on it.

Banks are absolutely considered victims, but if the victim didn't do their due diligence with respect to keeping their vault secure, then they absolutely share in the blame.

You know what we call banks who leave their vaults unsecured? Neither do I, but it's certainly not 'blameless'.

>When a teenager smashes in a storefront window, do we say they should've had bars over it?

Depends what's behind the storefront windows.

On the other hand I can certainly agree that we stop with the technology analogies before a car analogy is let loose and someone gets hurt.

It's pretty simple: both parties are to blame. PHP Fog did not do due diligence in securing their servers, and the attackers committed illegal and distasteful acts.

If I leave my front door unlocked and someone walks in, there are several possibilities. If they just look around and leave, or say lock the knob behind them and walk out, that is one thing. If they smash my furniture and knock over all my plants, that was their choice, right? That action is illegal and immoral regardless of the fact that it was my negligence which made it possible. We all know these dumb kids should have reported the vulnerability responsibly. That would have benefited everyone, especially themselves. They might have been getting job offers instead of bad reputations.

If you were walking down the street and my 9-year-old daughter were to run up to you and stab you in the eye, is it your fault for not wearing a helmet? Everything is penetrable given enough time, money and patience. I agree that PHPFog made some mistakes, but it's not like they were being willfully negligent.

Wow, never seen condescension clothed so well before. You should realize that by saying "PHPFog is the only responsible party", you are open to some perverse accusations (like, if your kid turns out to be a criminal, you and the victims are completely to blame).

He is 16 and SCARED. What do you expect? A slick PR campaign? Heck in some other countries, kids aren't even considered adults at that age.

He is naive and immature and realizes what he did is wrong. To make amends, in true 16 year old fashion, he gives a bullet list of errors to try and help undo the damage.

For his inability to communicate, show restraint, maturity, planning and foresight, for the very crime of being young and immature, he gets people wanting to throw the book at him? In that case, can we please, pretty please, torch wall street?

Its quite likely that he realizes that giving an error list is BAD and STUPID, and now is trying to back pedal by putting on a brave face to ignore the bone headed-ness of his (compounding) mistakes.

Try and imagine exactly how YOU would feel if you had a huge amorphous mob saying "The FBI should come for you"? At 16 you have NO scale in your head to cope with that.

Restraint is (one of) the hallmarks of maturity. As is intelligence and not taking good faith for granted - like not sending a list of errors you can be prosecuted for.

Here is what I would do - call this kid parents and Leave it at that. Let the family know how close he is to being in BIG trouble. If you want to do one better, give him a constructive outlet. He is already probably one very, very, very, miserable and frightened kid right now. And he should be.

Its called grace forgiveness and wisdom. As adults, we are supposed to have it. You are NEVER going to deter kids from being kids. So you need to ensure that they are scared and know where the line is drawn, so that they can become effective productive Adults.

Bah! Tomahawk 'em!

I know, they acted as if they could get away with it by apologising. However, whatever you think of PHPFog, a lot of people have invested everything they have in that project, and it could have (and still could have) done irreperable damage to their reputation and investment. Big companies can tank this crap but attacking a new startup is like punching a child.

Therefore you want to sue a child?

bring him to criminal courts. close enough.

I don't know if you were ever a teenager, but I remember when I was a teenager I had a willful disregard for consequences or how my mischief affected others.

I think they should definitely get something a little stronger than a slap on the wrist, but also remember that they're just teenagers who don't know any better. All they really need is enough of a punishment to learn their lesson, and they'll probably end up productive members of society.

Lucas and his team are amazing. Everyone makes mistakes, and no system will be perfectly secure. So in my mind the best parts of the entire post were these:

"We have hired professional white hat hackers with government level security experience to attempt regular pen tests on our system, both as regular users as well as giving them special access and seeing if they can get through."

and

"If you find a security flaw and report it using the Full Disclosure Policy to security@phpfog.com with notice, we will help strengthen your security reputation in a very public way and reward you generously."

Hiring white hat hackers is a joke. They are the ultimate in hacks.

Step 1. Open up Metasploit Step 2. Button Mash. ???? Step 3. Profit

I downvoted this too, but wish I hadn't. He has a point.

He could have made his point in a more intelligent way.

Just, do be aware that there are a lot of people that fit the description he gave; they are particularly numerous among the people who use the words "white hat" or "hacker" (with any modifier) in their services.

Downvotes because of phrasing? Here.

So called 'white hat hackers' tend to be fraudulent script kiddies who couldn't hack their way out of a gibs0n.

They often attend classes like this http://www.infosecinstitute.com/blog/ethical_hacking_compute... and read a book or two like this http://www.google.com/products/catalog?q=hacking+exposed&.... Some times they'll even have a sweet certification like this https://www.eccouncil.org/certification/certified_ethical_ha....

And at the end of the day all they're doing is getting the down low on your system with nmap and then going all turbo with metasploit. And if they are feeling up to a challenge they might even rip someone's exploit from milw0rm.

99% of them are frauds and the other 1% are sellouts.

Thanks for rephrasing. It really does make a difference. I actually got some value out of this comment, whereas from your previous comment I got none.

Can you explain your "1% are sellouts" comment? Are you saying that top-tier crackers that accept money for their services are sellouts?

I'm sure this is going to ruffle feathers, but whatever.

The way you become a 'top tier hacker' is by exploiting and reversing in the community (black/grey hats). Much of this is done through working with people who you respect and may have more experience than you.

Through this comes a certain level of respect, to where going public with exploits and such is a hindrance to the community and only serves your selfish agenda for fame or whatever.

Your exploiting all the people you've worked with in the past, ripping their ideas, and handing them off to assholes who then try to teach classes on 'ethical hacking' or whatever.

So yes, I'm calling them sellouts.

*Edit - Another problem I have is that these people generally tend to misrepresent 'black hacks' as a whole and try to play themselves off as bigger than life personas.

You're judging a point by the way it was made rather than its ultimate goal?

I mean, I'm all for praising gymnastics, but if we're trying for truth here shouldn't endgame be valued over execution?

Nessus?

The problem with pressing charges as a detterrent is that it is fundamentally unjust, because the punishment is set up as a detterrent and is disproportionate to the crime.

Example: imagine the country of Dictatoria where if you jaywalk you are publicly tortured for a couple of weeks and then put to death. "As a detterrent"

A little extreme? Well, consider the 10-20 (or more) year sentences for cyber-terrorism these teenagers are going to get if the FBI throws the book at them.

That kind of law enforcement agency doesn't understand the concept of restraint - they are set up to go for the kill, for the maximum charge, for the maximum sentence every time.

You can lay charges thinking to give them a slap on the wrist, but the steam roller that gets set in motion is designed to crush them flat with no mercy.

Why do you think these kids would get 10-20 year sentences? Have you ever heard of a computer crime getting that high of a sentence? No one on http://en.wikipedia.org/wiki/List_of_convicted_computer_crim... has gotten more than 5 years, and they include people who have sold access to botnets of hundreds of thousands of machines. A quick Google search reveals someone sentenced to 20 years, but for stealing millions of credit and debit card numbers, which is actually a sizable crime with significant damage to many victims. http://www.scmagazineus.com/hacker-albert-gonzalez-receives-...

There is a story along these lines, it happened recently in New Zealand.

Some parents had an argument with their teenage son, and he went off in a huff taking the family car (technically without asking permission). His parents thought to themselves "I know, we'll teach him a lesson", so they reported the car stolen. They intended to later on drop the charges (which would have royally peeved the police, making a false charge is also a crime).

Anyway, the cops caught the kid, and because Grand Theft Auto (or whatever the NZ equivalent is) is rated as amongst the most serious crimes, he got sent off (before formal charges were laid) to a maximum security facility in the back of a paddy wagon.

He didn't even make it to the maximum security facility. One of the other prisoners being transported there in the same van killed him.

Sounds like you're talking about Liam Ashley but have some of the facts twisted:

* it wasn't recent (2006)

* he was sent to prison because his parents denied bail, not because GTA is some heinous crime in NZ

http://en.wikipedia.org/wiki/Murder_of_Liam_Ashley

I would encourage them to pursue legal action. I think it's right, justified and also strategically important—you can't have other kids thinking they will get away with it in the future.

Security has nothing to do with feelings. Given the amount of should-haves in their explanation this should be a welcome wake-up call to them, a free audit. Other than playing the victim card, they handled it quite well.

Nothing better to sharpen your skills and higher your standards than a _good crisis_

Hire the hackers. It's what the CIA would do.

Hiring the hackers is a terrible idea. If they had done it ethically, yes. But do you really want to hire someone who has already displayed highly unethical behavior, and is likely a ticking timebomb? There are plenty of smart, ethical hackers out there.

I don't think you can draw those conclusion from the evidence presented. I would guess that quite a lot of the top white hat hackers today sharpened their skills early with some highly unethical behavior of their own.

A good example of this would be how Chris Putnam got hired at Facebook.

http://www.quora.com/How-did-Chris-Putnam-get-hired-at-Faceb...

Oh wow, I just realized that Chris Putnam is the same guy who used to post on the Something Awful forums back in 2003-2004 or so.

I remember him getting trolled out of there after showing any kind of ambition beyond posting on the forums. Same thing happened to the guy who started Imageshack (originally an image host for the SA forums), Eli Hodapp (who later became one of the main TouchArcade writers), and probably some others I don't remember.

Well dang, good for him!

Damn. I had no idea that being run out of SA with your tail between your legs back in the day was such a predictor of success. Who'd have thought it?

Only if they demonstrated great skill - which I don't think happened in this case (but I'm no security expert either)

And great restraint/maturity. Which didn't happen either.

Or do like the FBI and offer them a job, when they come for the interview they get frogmarched into the police cruiser.

Valve did that, the FBI was just the tool that was used for the job.

I'm struck by the similarity in some respects of the two cases. I would guess that the same people that got really upset about Valve doing that would be a bit saddened by this, too.

Yup, ruin a kids life to act as a deterrent to other kids. Fortunately this strategy has been keeping our products protected and secure for years after Valve had the FBI track that kid that leaked half-life 2's source code and put him in prison.

Unfortunately, this situation was much more severe than your average multi-million dollar AAA gaming title being leaked before release (http://www.ea.com/crysis-2/blog/crysis-leak ), so this requires the FBI and prosecution.

This isn't a DRM break.

The blog post is riddled with the words "luck" and "timing" which brings doubt into my mind that the team can actually take full responsibility for their actions.

"aware of the potential security threat " but they left it for the next week, who honestly here would do that?

I have also seen comments around the web of migrating to Php Fog because of how they handled the situation. If you are one of these people please enlighten my mind as to how you came to such a logical decision or how much you get paid per year.

Also if Php Fog could enlighten us on how their terms of agreement will work in the case where our intellectual property is stolen on no fault of our own.

Save your sympathy for the sites that are still down, four days and counting

I couldn't agree more. The phpFog team cut corners to deliver quickly. We (devs) all do it. The important part is to clean up after yourself.

The whole blog post seems a bit melodramatic. I mean seriously, who here hasn't spent 3 all nighters in a row fixing a mistake? sack up and do what you should've done before deploying other people's data.

...and who would seriously sue these kids? they handled it poorly but they're smart (definitely smarter than i was at 16) you're lucky it was curious kids, rather than malicious (and experienced) hackers that would've been harder to catch. Do you really want to burden them with a criminal record for life?

Am I the only one who becomes functionally useless after the first 24 hours? I'm nearly 30 now, but the maximum I'd have done 10 years ago was 36 hours.

Exactly. This post tries very hard to trivialize the security holes, and blame their problems on bad luck. They had problems because they decided not to fix glaring security holes immediately.

My understanding is that this is all caused by an unsecured failover server. Hopefully, we get a bit more details of how this came to be and learn what they intend to do with future server deployments.

  "aware of the potential security threat " but they left it
  for the next week, who honestly here would do that?

Just about everyone. There are always 'potential security threats' that are deemed unlikely to be exploited and that you therefore do not give priority above the multitude of other tasks you have to do. They took a chance and I don't doubt everyone here does that on occasion.

I am bothered by some of the language in this post:

- we were aware of the potential security threat behind post-deploy hooks and were about to disable them [...] but...

- we were days away from replacing this server

- They were a short-term stopgap measure we had been planning to replace

To me, it sounds like the real problem could have been stated as "We were lax on security," but almost worse than that is the lack of accountability that I sense from company. Yeah, maybe it won't happen again, but it's hard to be full of confidence to buy into a service like that.

They seemed to be blaming it on "bad timing" as if these things were ever excusable. These are also things that you either do or don't do. Your systems are either secure or they're not. "They were going to be secure tomorrow" does no one any good. It doesn't look like any of the parties involved learned much of anything from this episode.

> Your systems are either secure or they're not.

[citation needed]. Security is never binary. No matter what security measures you take, there are always zero-day exploits, social engineering, physical access, heavily-researched-and-highly-targeted attack vectors, etc.

Security is the opposite of convenience and accessibility. The right thing to do is to analyze what you are trying to secure and ensure an appropriate level of security proportional to the sensitivity and business impact of the potentially-exposed system.

There's no such thing as "secure." It's a continuum and it's always a tradeoff. Would you spend $5000 to protect something that's worth $50? It sounds like this site was in beta mode, and they made an understandable decision to focus on building the product and growing a customer base in lieu of ensuring top-notch security. In retrospect it was the wrong decision, but you don't hear about the companies who follow this approach and don't get publicly hacked. If they spent all their time on security from the outset, they wouldn't have anything to protect.

I read him being very apologetic for their security shortcomings in all of the appropriate places, and only blaming delayed fixes on timing issues. He was very contrite and forthcoming about their security issues. Accountability was all over the article.

I disagree. Lucas names these children and attempts to personify them so blame can be shifted to the "bad guys" rather than his company.

Lucas's post does not say "We screwed up." He says "We got screwed by Elliot."

I'm saddened most because Lucas is not embarrassed to point out he was outwitted by children.

When I foul up at my job I don't send an email detailing how some nasty client did something. I summarize what went wrong, how it should have been prevented and what steps I will be taking to prevent it in the future.

I would never write an email:

James Smith, a really evil customer (who happened to be working while there was thunder and lightning like Dr Frankenstein!), decided to try system("rm -fr /"). I knew it was possible, but I didn't feel like fixing it. Also I didn't feel like securing any of our other systems which explains those tweets, blog posts, DNS changes, and email compromises. I was lazy, but It's not my fault.

gg, parfe

P.S. Credit cards probably didn't get compromised. Tim the intern was the one who implemented the payment system and he had his own passwords set.

(Note: I move this comment as I replied by mistake to CGamesPlay.)

Did you read the article?

Lucas's post says: "This was really naive and irresponsible of me." That doesn't sound like he's shifting blame to me.

You say: "I summarize what went wrong, how it should have been prevented and what steps I will be taking to prevent it in the future."

The article is essentially just that, with one exception; they didn't list steps they "will be taking" to prevent it, they listed steps they have already taken in the last 3 days.

As for Credit Card: "Credit cards – We have never stored credit cards on any PHP Fog server. There was never any possibility that credit cards could have been compromised by this attack."

I didn't even make it through the headline before being concerned... particularly the part saying "Why it Will Never Happen Again".

I mean, yes, by all means implement measures to avoid this sort of thing from happening in the future but "It Will Never Happen Again" is a very, very bold statement on security. The kind I associate with people who still don't really "get it".

If I was a customer of theirs, I wouldn't have really been (too) bothered about the initial intrusion. However, hearing them say "Why it will never happen again" would make me switch providers. In my mind being willing to say "it will never happen again" implies a basic misunderstanding of the security environment and is tantamount to a guarantee that it will, in fact, happen again - perhaps even regularly.

The article starts out in that tone, but it changes to be pretty remorseful after that. For example the "why it won't happen again" part.

I agree with parfe's comment below.

They sound incredibly laxed on security and the "we were days away from fixing it" could be complete bull. To Lucas, it probably sounds better to say they were close to fixing it instead of admitting they were unaware of these exploits.

I find the disclosure in the blog post great, but the conditions they had leading up to the hack very disappointing.

If they were aware of the exploits, they should have taken quicker action. They'll probably be focusing on security big time now... they have no other choice.

i felt they apologized rather well. its difficult to apologize and explain what happened at the same time without sounding like you're making excuses or trying to skirt responsibility.

"We have hired professional white hat hackers with government level security experience to attempt regular pen tests on our system..."

I guess whenever I read this kind of statement from now on I'll be thinking of HBGary and chuckling a bit inside.

At the risk of that comment being taken as a joke, I've done a lot of work with the federal government, and I can assure you that while the level of hilarity that HBGary has generated, the typical level of talent in government cleared individuals is not necessarily great.

I don't mean to impune the capabilities of the people involved (I don't know who they are,) and it isn't to say that you can't find some AMAZING talent in the government realm, but as in all fields, it's the exception, not the rule.

Reminds me of when I was in the Navy and I went to Navy Security and Vulnerabilities Technician school. I was all excited, so I went out and bought a copy of Hackers Exposed and read through the whole thing, learning everything from how to determine what family and version of operating system a computer is running by what ports are open, to how a buffer overflow attack actually works.

Fast-forward to the class, and we're sitting there running tools like BackOrifice that exploit vulnerabilities that had been patched for years, and learning that a SYN flood is "a malicious attack". That's it, just "a malicious attack". When I asked about the difference between a SYN flood and a Christmas tree attack, I got a blank stare and "they're both malicious attacks".

I spent the rest of the class in the back of the room, reading the Armadillo Book.

Also, I did not once in my brief Navy career get to hack an enemy computer. Hugely disappointing.

Yeah, military schools on scientific/technological subjects can be very disappointing. Not all of them, obviously, but computer science topics get distilled down to be accessible to the bottom 20% of attendees. Having spent over a decade in the Marines, I learned not to get excited about anything like that... unless the schools were taught by civilians.

That's exactly it. I've worked for agencies where the Chief IT Security Officer (if they have one, or an equivalent role) got that exact training, and nothing more, and was considered the site expert.

Like I said, it isn't all bad. Two of the best security guys I know work in the government, and one of them was actually ex-Navy. But the hurdle for finding people that can get top secret clearances AND tie their shoes often proves too high to hire anybody, much less somebody qualified.

Hacking Exposed: http://www.amazon.com/dp/0072121270

Armadillo book: http://www.amazon.com/dp/0596003439

I mentioned this last time, but I don't think anyone was interested, but the "John" guy is compwhizii (same handle on Twitter) who runs the forums (facepunch.com) for garrysmod, a very popular game. I will be curious to see how garry (owner person) responds to this, or if he already has.

Elliot is apparently VERY scared and blames John (compwhizii) (edit: not john, he blames someone else called supersnail1): http://www.facepunch.com/threads/1071855-A-member-of-Facepun...

Here is (compwhizii) Johns reply: http://www.facepunch.com/threads/1071855-A-member-of-Facepun...

And here's Elliot's "official statement": http://elliotspeck.com/phpfog.html

And for anyone who missed it, here's what Elliot posted in the previous HN discussion about the phpFog breach: http://news.ycombinator.com/item?id=2346161

Hi, I'm <full name>! My site says what city I'm from. I've written publicly to admit that I committed multiple crimes, definitely without consulting legal counsel first! I even put them in a nice bulleted list that can be copied and pasted right into a complaint. They're pressing charges, but that's bullshit. I'm 16!

Not too bright, are we? Instant message the company you just hacked and bust out from behind your handle, then provide evidence for the prosecution in the form of a Web page? What is with kids these days?

It only takes one episode of Law & Order to figure out how to proceed here. Clue: Attorney.

Somebody really needs to get in touch with that kid's parents and let them know that they need to find the mute button on their kid and retain counsel.

After reading that I think it's a pretty strong argument against those claiming adult status for him. He clearly doesn't understand the situation he's in or the second order impacts of what he's done.

As soon he says "I don't believe I did a bad thing" I thought this boy needs to be prosecuted.

Personally, I'm a fan of some of his post-hoc justification:

"Following this, I took a hold of their Twitter account and posted a couple of bits to draw attention to the fact. This did two things. One, it showed people the system was insecure, but on the other hand people always subconsciously root for the underdog; I drew attention to the company and the product. I know a number of people have actually registered (or intend to register, registration is closed) for phpFog since the incident thanks to the attention drawn to it by myself."

The last bit is priceless. The mere fact that the kid doesn't show any sort of adequate moral judgement should justify prosecution.

Isn't the lack of adequate judgement (moral or other) the definition of being a kid?

At least he could be more of a kid and pretend. Which means I actually agree with you.

Before I realized he'd posted those comments 2 days ago, I seriously considered trying to track down his parents. Do you think they even know? Their kid is arguing with people on message boards about a breach. And not arguing "I didn't do it". Crazy.

Search hard enough, and his phone number/address is available if you're really like to.

However, I won't post them here. That would be irresponsible disclosure. ;)

Elliot's statement and a lot of his Facepunch comments have been taken down. As an educator, I found a lot of the comments there pretty alarming... and want to ask everyone here to consider a few things when posting about anything related to kids.

Kids (and like it or not, that IS what they are) are notoriously dramatic and these kids are definitely experiencing this as a crisis, no matter how they present in their comments.

They are also now being border-line, and in some cases overtly cyber bullied by some of their peers online which may be devastating if they don't have great face-to-face support. As adults, we have a responsibility not to add fuel to that fire and remember that if teenagers were predictable and transparent, there wouldn't be so many tragic cases in the news where they hurt themselves and each other.

Part of the problem is that teenagers and young adults aren't great at predicting consequences. This isn't opinion - it's science.

Especially in males, the part of the brain that is responsible for predicting consequences is not fully developed until the early to middle twenties. This is a double whammy because it leads to their bad decision making, but also to their belief that when something bad happens as a result, that it's literally the end of the world... that there is no way out.

There is no disputing that what these boys did was a big deal and that there have to be consequences. But it's also just a symptom of a bigger problem: as a society, we have failed to keep pace with the challenges that are the result of the first generation of digital natives (kids) being raised and educated by a generation of adults who are (at best) digital immigrants.

This post is getting way too long so I'm going to finish it on my blog @ thenewtag.com But thanks, criticsquid and nbpoole for posting some of the aftermath. I hope that one of the 1st things phpfog (or the authorities) did was involve these boys' parents?

Reading through those forum posts, I have less and less sympathy for these guys.

The phpfog guys really deserve praise for being so open on this issue. As a fellow engineer, being able to learn from their mistakes and see exactly what they could have done ahead of time to avoid the disaster is priceless.

Just goes to show that those with the time to spend are the most likely to break your stuff, even if you pay "professional white hat hackers" to test your system.

On the contrary, they knew of security vulnerabilities and intentionally left them unpatched, then blamed it on chance and timing when they got owned because of it.

Avoid phpfog if at all possible, in my opinion.

If I lost $1 for each time I left a known vulnerability unpatched because I was convinced I had more important work to do, I would be a very poor man indeed.

Honestly, there are very very few developers that fix security problems in beta environments before anything else. In my experience, it's more likely that you're fighting fires, handling outages, and dealing with problems of scale than fixing security vulnerabilities.

Besides, isn't a beta the correct time to find these security issues? (Design / Alpha would be the ideal time, granted, but sometimes that's not possible.)

It seems like incredible coincidence that allowed this to happen but when I think back to all of the security incidents I've been involved in, it always seems this way.

I guess the best way to think of it is that badness on the internet is like water. It will flow into every tiny crack in your wall you haven't sealed up tight. A crack in a dam doesn't leak less because its in an "obscure" location.

Goes to show you why the DRY principle (I might be stretching that analogy here, but bear with me) is important here - if you have old stuff lying around in production that was cloned a long time ago, you might forget about it and open yourself up to unfortunate incidents like this.

PHP Fog is doing great work to make the PHP ecosystem easier to work with, and I hope they didn't suffer too much from this mistake.

Still stretching the analogy, but the same could be said of their password reuse.

That's exactly what I thought of.

DRYP - Don't repeat your passwords.

I don't think DRY applies here unless you really stretch it. The "fix it if you see it, don't put it off" principle fits a lot better.

While it is admirable and good that they have learned from their mistakes and are taking steps to reduce the likelihood of getting hacked in future, to say "never again" is to paint a big red bullseye on yourself.

Wait...their model is an EC2 instance per customer? The normal limits Amazon imposes are 20 reserved or on-demand instances and 100 spot instances per region. You can request more, but will Amazon really accommodate a one instance per customer model?

Amazon is happy to. The limits you cite are merely the point at which you need to have a conversation with Amazon staff. They are quite happy to accomodate _much_ heavier usage from customers.

When I asked for a raise to my limit they denied me, on the basis that my usage was insufficient. Of course, my usage was low because I hadn't launched my product fully because I didn't have enough instances to serve a lot of customers ... catch 22.

So I had to build out a rather convoluted architecture that used the loophole of deploying to multiple regions and failing over to whichever region would give me an instance ... which gives me up to about 80 instances ... just barely enough for me to get going with a trial beta program.

Which is all just to say, it is slightly more than just a "conversation" that you need to have to get a higher limit.

Sorry you had a bad experience. I have gotten nothing but superb support from Amazon staff. I'm surprised they weren't willing to accommodate you, doubly so if you were ready to pay.

One thing that surprises me is when people talk about utilizing multiple availability zones in EC2 as some sort of burden. It's very clear from their documentation and architecture that you need to be capable running in at least 2 availability zones regardless if you want any sort of availability.

> people talk about utilizing multiple availability zones in EC2 as some sort of burden

My use case isn't for an ongoing server where you require availability. It's purely about compute power - I don't care where the compute power comes from but preferably I want low latency to my customer. So ideally I would just get all instances for any given customer from a single region.

I did find in the end that, as you say, I would sometimes not be able to get an instance in a region even when I was below my 20 limit for reasons internal to Amazon, so the failover work was going to be something I had to deal with anyhow ... but it just added complexity to my life earlier than it would have otherwise.

Edit: I would also mention that I certainly don't think of it as a "bad" experience. I think it is something of a small miracle that Amazon offers the service they do in the first place and I certainly understand why they have caution about handing out large limits to just anyone. I only made my comment above as a kind of caution to not just assume you're going to get a raised limit from Amazon immediately and especially don't leave talking to Amazon about it until the last minute if you're planning to launch something.

Thanks, that's an angle I haven't seen since I've mostly used EC2 as a hosting service. What little batch work I've done on it hasn't involved instance counts where I ran into limits. (Now, EBS i/o limits and other things... :/ )

"Can I please speak to your supervisor" works sooo often in these situations.

I've always assumed that the limits are there to make sure that no one can order 1000 instances and pay with a bad/stolen credit card. In other words, if the customer does a dine and dash, how much are we willing to lose?

Leaving the doors to your house wide open does not grant every passerby the right to enter.

So, yeah, PHPFog screwed up and did that. Then these kids went in, threw paint on the walls, smashed some windows, etc.

PHPFog was stupid - they admitted that.

The kids were criminal.

The first is not illegal - the second is.

What a crazy story. If the timelines are accurate there was an extremely small chance of this happening. Bad luck all around.

My site is still down, guess i'm in the unlucky 1%.

Yeah but the problem with 'things planned for the near future' is that they have a tendency to stay in the future until something like this happens.

Yes, and after the fact, they sound like excuses (even when they are true and they are not meant to be excuses as in this case).

Murphy's extended laws:

* It will go wrong at the worst possible time.

* If there are 4 possible ways for it to go wrong and you prevent all of them, it will go wrong in a 5th way.

Can you give us more details at help.phpfog.com please? (apologies if I'm breaking HN etiquette)

You don't hear about the 99% of the attempts that were successfully foiled by good coding practices/security audits, so given that we are hearing about a security breach at a competent firm, the likelyhood of it being a black swan event is probably fairly high.

You also don't hear about the attempts that were successful and unbeknownst to the hacked service provider.

Ugh, you shouldn't try writing an apology after not sleeping for days. Sleep on it first, always sleep on it. Talking about prosecution and explaining this with a framing that it was all a fluke caused by the only person who was silly enough to IM you with a confession... add one more person who will never be a customer of yours with an apology like that. Now I know you're irresponsible.

Seriously don't write official blog posts for your company while you're experiencing "I was just in the field for days trying to fix this stuff" emotions.

Calm down, then try and be graceful about the fact that you were hacked by a few clueless kids. (Clueful kids don't let you know who they are.) Then try and figure out how to protect yourself against people with a clue.

Wow, that is quite the list of security measures that they had almost but not completely/correctly implemented, or hadn't got around to yet.

I guess the real moral of the story is to finish what you begin, or don't keep putting security off until it is convenient for you.

Never? I would be cautious about issuing a challenge like that.

They might still have a security hole big enough to drive a freight train through, but that specific attack will never happen again given that they've shut down the shared failover server.

Congratulations to PHPFog. They've managed to direct the attention to the 16 year old kids rather than their own incompetence.

Is it me or no one mentions the lack of expertise of the PHPFog team in PHP and Systems Administrations.

Sure kids broke in and the way they published their findings was despicable. The fact remains that PHPFog was utterly broken to pieces and the exact essence of the problem is simply the lack of knowledge in their field.

I am very disappointed by the tone of the blog post and think PHPFog don't really have a notion of what they are doing. I would much rather seem them where they belong, in the Ruby world where their experience is.

Their response and abilty to turn the situation around is a case study in dealing with a difficult situation. Kudos! I'm saving their response and will use it when dealing with things. Being able to have a counter party to identify has definitely helped in handling the situation. I didn't realize how powerful that can be until I saw this, I learnt something new.

Its a brilliant piece and a great start/way to restore faith and recover from what must be a pretty grueling ordeal. Good job.

Great to see disclosure. This can happen to anyone, and more so for startups, where labor is short, focus is on developing features. Using the phrase "Never Happen Again" is a bit strong though. Security is risk management; spend until you can accept the remaining risk while still maintaining profit and avoid being a hacker's low-hanging fruit.

This post convinced me not to use PHPFog. They reveal more in their lack of foresight and security prevention measures than their response to what was otherwise a fairly trivial exploit. I am not sure this blog post was helpful in convincing customers like me that want to feel that their infrastructure providers are on top of things.

Here's an interesting tweet from one of their developers.

http://twitter.com/ReinH/status/50348989366796288

> Your password in the database is SHA512 encrypted, but we're not taking chances.

I hope he knows what he's talking about and is just tired from the past few days.

Let's hope they are salted and iterated.

Or swapped over to Bcrypt

we've cleared the old SHA512-salted passwords out of our production database and have upgraded the password hashing to bcrypt, with a cost of 10.

Good call :)

Just in case anyone doesn't know why Bcrypt is so awesome, it's because it actually takes longer to hash (based on the difficulty level you set, and you can bump up the difficulty level as hardware gets more powerful).

For other applications, you want hashing to be fast. But for passwords, you want hashing to be as slow as possible without compromising user experience.

I wanted Lucas to link to Coda Hale's post on bcrypt (found by googling "Coda Hale bcrypt") in the blog post, but he edited that out. So it goes.

I remain in two minds about idea of charging the kids.

There is no doubt they did some things they should not have. And I don't doubt there can be a decent case built against them. But as someone who actually had something from his teen years come to bite years later, it's not pleasant. At least in my case it was a MAJOR maturing moment(also the worst day of my life). May be it will take a lawsuit to get these kids to mature up...to that extent anything that gets em to mature up before they really get screwed would be fair.

I'm not merely advocating another chance but actually something that gets these kids to be a tad more thoughtful about their actions. It's not always easy to do that when you are 16 and full of adrenaline.

I am sure, many of the HN users here would have found at least a loophole in similar systems in the course of time. What I do in such situation is letting the service know about the flaw. Isn't that the ideal behaviour ?

> Eliminate shared hosting failover server – We may never do shared hosting failover again if we can not guarantee its security. We might do a non-realtime failover to automatically launch a new instance for you, but this experience taught us what a bad idea this can be.

What does realtime mean in this case? Anyway, this isn't the only option. They could keep a few bare instances of their php stack online and simply run the deploy script instead of the image creation script. That ought to be able to run in under ten seconds I think.

Realtime means a request in flight is retried if there's a failure. Said a different way, the load balancer already knows about a spare at all times. This is compared to a ~5 minute downtime if spawning a replacement EC2 instance is required.

Nice idea on the hot spare instances however.

This feels like a business model where the lean/MVP approach isn't quite appropriate. A lot of things fall out of that decision, not the least of which is that the exposure surface area you get from an environment that allows user-sourced code on purpose is enormous. I feel for the guys going through this but there were a lot of errors in the wild all at once to allow this to happen.

IMO, in this sort of business it is important to define the right MVP.

There is no such thing as bad publicity! Kudo's for turning lemons into a viral blog post! Although, if I understand correctly, you were reusing passwords and storing them in plain text! This is an ABC123 computer security nono. Thank goodness it was just some young script kiddies and not someone with malicious intent!

great to hear all the details so quickly so that others building similar systems aren't in the same situation. as fellow PHP'ers its also great to hear that you are not blaming it on PHP somehow (no fuel for the php haters).

  2:56:45 AM Elliot : then I used the method detailed by turby
  2:56:46 AM Elliot : to gain root

Has anything been said about what this method was?

If it's what was on the PHP Fog Sucks website, it worked something like this (I may be misremembering a step):

1. Use the post-deploy hook to chmod /home/ubuntu/.ssh so that it could be written to.

2. Upload a PHP shell, use it to write your public key into /home/ubuntu/.ssh/authorized_keys, and get the public IP of the EC2 instance.

3. SSH into the box, sudo su will get you root.

Ok, thanks. I was wondering if there was some other root exploit at play, not just bad system configuration.

So, shouldn't the first thing you learn as a hacker include how to mask your physical location so as not to have the FBI knocking on your door?

I feel for the people at phpfog.com, but this is a bigger blow to cloud computing.

Customers who are already pretty risk averse to their data being stored in the cloud would see this as another reason not to take the risk.

The cloud computing consortium needs to work on a stable stack as well as figure out how to audit that it works properly.

In addition, it calls for security ahead of features. Given that phpfog is funded, they'll need to implement the equivalent of a bleeding edge stack and a locked down stack.

Deleted comment

They're actually a Ruby shop according to the leaked codebase.

Actually phpfog is built on Ruby

php... a language by amateurs, for amateurs. phpfog... a service by amateurs, for amateurs.

Your argument could also apply to the Linux kernel.

I know the point you're trying to make, but it's a weak one.

75% of the kernel is written by corporate employees.

http://apcmag.com/linux-now-75-corporate.htm

True, I don't know what percentage of contributions to PHP are made while in employee of a corporation funding said contributions, but there are commercial supporters of PHP out there. I'm surprised how few people don't know that Zend exists.

The original comment just struck me oddly because I've come to assume that a large portion of the software that I use regularly was originally developed as an amateur project, or by amateurs.

75% of the kernel is written by corporate employees.

And that implies what about the kernel?

I'm not sure where you're heading with this. I was trying to tell the parent comment that arguing against 'amateur' coders writing php is the same as the Linux kernel was factually incorrect.

I bet most contributors to php are corporate employees too

And we're all aware that this wasn't a PHP exploit, right? Right?

for all the downvoters, it's pretty undeniable that as far as security is concerned, it was amateur hour at phpfog.

If you wanted to get that point across, you could have said it much better. Your analogy to the PHP language itself is completely false and not in any way useful.

Reply on news.ycombinator.com