Rely your web startup on Rackspace Cloud? Think again
bencheng.net
bencheng.net
That said, it's good to have a reminder of the risks of outsourcing your hosting. I still think the tradeoff is worth it for experimental products where you don't want to invest too much upfront.
This would be analogous to, for example, AWS taking reddit offline because a user posted a phising link.
Nearly every web business which lets people put information on the web that others can see will face abuse issues at some point or the other - and cutting off a business and its legitimate customers because of one client misusing the service does not inspire confidence.
Many comments here take the phishing as "a fact", their terms might grant them the power to shut any server down but this is a threat I think every startup should learn if they use Rackspace Cloud or consider to. And we learnt that.
24 hrs is not just for respond to remove the content, it is also for the server providers to verify the complaint and react responsibly.
I don't think it is possible for few-man startup can responds in 1 hours for 24x7. I would choose to use an alternative hosting that give a longer gracing period.
Unfortunately, during that grace period, numerous people may be receiving spam emails directing them to the site, and some of those people may be naively entering their information ...
I really dislike the way most service providers and the like handle spam, but unfortunately, I too must side with Rackspace on this one. They simply can not afford to "wait and see" until the site owner responds, or provide a grace period while the site owner tries to figure things out.
Phishing attempts must be handled by site owners as though their server has just been compromised and someone is currently downloading the entire password database: the server must be shut down immediately, the problem fixed offline, and the server only brought back online once the issue is fixed.
Sorry. :-/
1. Keep the very short notification period but also try to reach the site owner via phone or IM
2. Lengthen the notification period if using email only
(Note that I have no problem with short notice and email only if the customer was given the option of providing an emergency contact method but chose not to, and that I otherwise generally agree with the response.)
It seems like the real flaw here is the combination of lack of communication and lack of warning.
For your argument, I just created an wufoo form which should take down immediately once discovered. http://rickmak.wufoo.com/forms/phishing/. IN that case, I am sure only my account will be taken down, not the whole wufoo.
Actually, it depends on size. If someone created a phishing site on Heroku's, Amazon probably won't shutdown all Heroku sites. But to let Heroku to investigate. For small startup like pandaform, no luck. Rackspace just regards you as one site.
Pandaform can handle things better, like banned "password" field like wufoo do.
This is the kind of thing where a customer who gets their information stolen while Rackspace is waiting for the grace period to expire might have a legal cause of action against Rackspace.
Ultimately, I think Rackspace did exactly the right thing here. If you are operating a service that would potentially allow fishing, then you are bearing the risk of policing your users. Asking Rackspace and affected users to give you a grace period is asking them to bear the risk instead. I 100% agree with the decision to immediately shut the site down.
If Amazon got complains about Heroku then I'd certainly expect them to be investigated, and in Heroku's case I'd expect Heroku would take over and shutdown the phishing site.
Yes, if there are enough complaints and harm that may come from it is serious enough.
A second complaint, without any investigation, would result in the termination of his account and destruction of data.
That is not reasonable.
We don't know this. We have no idea how many complaints rackspace has against this guy. It could be one or it could be dozens.
Once a phishing form is “in the wild,” every minute counts.
The burden is on the service (your site) to prevent or quickly act to rectify a situation, but if your provider determines that it must intervene, then it is well within it's right to.
Also pandaform doesn't allow use to put any script or password field in the form, which the quality of the "phishing" form is not as serious as what we thought as a normal phishing site do.
I'm sure if Pandaforms had done this (which is difficult when you're a much smaller startup than Heroku) then their server would have been left untouched.
You can argue that Heroku would have most likely got a phone call and that Pandaforms deserve the same treatment, but I don't think that they'd have been allowed to leave phishing sites up for any period of time without their servers being placed in jeopardy either.
Try your best to get Robert Scoble's attention on this. Rackspace has tasked him with being an evangelist primarily focused on startups and bringing them into the Rackspace services. It's my understanding that things like this should be his primary concern.
I could be mistaken, but I'm used to that.
third - second paragraph is my interpretation of a conversation i had with some people from rackspace and it could be wrong.
also, hello from minneapolis.
If I lease a server from linode or AWS or theplanet or serverbeach or ${your favorite hosting provider}, would the situation be any different? I understand the article's author frustration with Rackspace, but it's a single data point hence hardly enough to be a basis for an intelligent choice of hosting provider.
I'm not even sure if I sympathise with him. You can argue whether 1 hour notice before disabling a server is enough or not but there is an obvious conflict of interest.
The interest of the person hosting server, who can potentially be a phisher himself, is for the site to stay up as long as possible.
The interest of the public is served by terminating the server as quickly as possible.
Everywhere, the more you pay, the more effort they will put in to helping you clean up your messes rather than shutting you down right away, which makes sense, because helping you clean up your mess is an expensive business to be in.
Perhaps Rackspace cares more about it's shareholders than customers?
"We talk AND we listen. We're big fans of transparency and feedback. Technology works so much better combined with a bit of humanity."
If you're going to pull the plug or even thinking about it... email simply isn't going to cut it. You need someone to call the owner and make contact to explain what's going on or how to resolve it. I've had my servers compromised, I've had phishing content setup before, I have never had the plug pulled. I've had hosts contact me, give me appropriate amounts of time to handle it and some of them even offered to help secure my box or look into how it got compromised in the first place.
I understand startups may not have the personel to react as fast as a larger company with dedicated personel, and I understand that there might be a very large percentage of startups on Rackspace which might account for a nice chunk of revenue.
But what makes startups different from my personal website or a larger corporation? If Rackspace receives a complaint about a phishing site hosted on their servers, they should do what they can to correct that, regardless of which client is using the server that has the phishing attack. The startup should get the same treatment from Rackspace as the large corporation and the guy with some simple homepage.
I do think that site owner of any scale IS responsible to abuse complains, but the scale of the company do make a different. The influence and harmfulness of a fake shop in Amazon is not same as a fake shop in a very-small-online-shop. Thus people expect Amazon to take action instantly.
A start-up is not possibile to react as fast as large company, so why shouldn't we give them a reasonable time for them to do their job?
Unlike the hypothetical disabled person, however, your startup, when used for phishing, is allowing real harm to happen to other people.
Your attitude suggests that a) you care less about your clients than about what's convenient for you as a startup founder, because b) you're small. But that's a very bad excuse, and a dumb one, at that. Clients don't care about your size. All they care about is the results your products give them. And that leads to another question: why should anyone buy into your services if you don't/can't act promptly and responsibly in response to problems?
And yet you seem to be getting angry about Rackspace for caring about the clients affected by such criminal activity further downstream from them.
I'd think twice before buying into your products, with an attitude like that. Rackspace appears to care. You do not.
If they hadn't it's possible law enforcement could have gotten involved, you could have been arrested, all your business records and source code confiscated and have been offline for months. If you'd been using your own hardware that might have been taken away, too.
There are plenty of cases where that kind of thing has happened before.
When a server is 'shut down' for phishing or spam, a firewall blocks all incoming/outgoing except for traffic to/from a pre-determined IP address along with the notice that the server is being quarantined.
Site owner/admin can then access the server, perform any investigations or deletions necessary, notify data center, then data center opens traffic again.
Alternatively, something like a web-based shell allowing access, but all other traffic denied, would be acceptable.
I've had servers shut down for 'abuse' which was one complaint from someone at 1am local time for me. I supposedly got a call from xxxxxx at 2 am, notifying me that action would be taken, and my server was taken offline at 3am. I wasn't awake until 7am, and couldn't get things resolved until about 10am. I was told I needed to 'rectify the situation', but how can I do that when access to the server is blocked? It's a ridiculous execution of policy, and only serves to heighten everyone's frustration. A private web-shell or single IP in the firewall to allow access would resolve most of the ill-feelings site owners caught in this situation have had.
The problem is that there is no way to 'clean up' after a break in without booting from trusted media. Otherwise, there is no way to know if you have closed all the backdoors the attacker left.
Lesson learned? I think you should never have any operations where you don't have full control. This still holds true but it was a bit ironic because to have full control I ran everything on the VPS, effectively transferring full control to the VPS provider! It was the single point of failure.
Now I will transfer each different service to a different provider. Email to rackspace, websites to a host, git services to github and so on.
I think verification of the legitimacy of a complaint should be a critical step before disabling a site, otherwise you're prone to DoS.
It would also be good to know what steps the complainant took. Did s/he try to contact Pandaform, or immediately go to Rackspace as the owner of the IP?
Without knowing whether the complaint was legitimate, and what steps Rackspace took to verify this (or not) its tough to say whether their actions were appropriate.
The majority of commentors here assume guilty until proven innocent.
In fact, the author of the article has still been unable to identify the "phishing" form. Was there even really a phishing form?
With all this anti-phishing technology built into modern browsers, why is it Rackspace's responsibility to "protect" users from "alleged" phishing sites?
For copyright, we have the DMCA (for better or worse). Perhaps we need some similar sort of due process for hosting providers or cloud providers.
Most appalling is Rackspace's lack of transparency in handling this.
Pandaform was never contacted by the complainant. The complainant only contacted Rackspace. Rackspace assumed, without investigation, the complaint was legitimate and gave the guy notice via email.
Also, the real rub, is the fact the Rackspace would terminate his account if he got a second complaint.
So, it is not necessary to have an actual phishing form on his site to have his Rackspace hosting terminated. Someone simply needs to allege this to Rackspace, and his account and data will be gone forever.
That is fanatical? Again, I cannot recommend Rackspace.
That said, I think especially for higher-priced services, a phone call would be nice. (Note: I don't call my customers, though this is a policy I've considered implementing.) I'd be interested in what other people think about other notification systems.
Now, if I was a 'fully managed' provider that had a login on all my customer boxes, in that case, assuming the problem was a customer of a customer phishing rather than a compromised server, it would be fairly easy to log in and fix it. But that's not how I'm set up.
Should that distinction make a difference?
Still, I think it is a good idea, and one I ought to implement in my own service. The problem is that we all hate dealing with the phone, but that sounds to me like a lazy answer.
On the other hand, really, if you are an abuse problem, honestly, I don't want you as a customer.
It seems like they should have been doing some sort of verification of the UGC. Alternately, if they really wanted to go with a fully laissez-faire no-security-checks approach, they shouldn't have picked a hosting company (in this case, Rackspace) that requires customers to sign an agreement that has strict anti-phishing rules.
Some may argue having your own hardware is more expensive to maintain, but there is a definite advantage to controlling your physical hardware.
Rackspace is helpful until you have a real problem, and you are left to fend for yourself.
Did the customer simply set up a form that asks for a user's email address? because if so that describes tons of other services out there... e.g. Wufoo, Google Documents (but I guess they are not hosted on Rackspace!)
Look professional? ;) Similar form on pandaform will make pandaform shutdown entirely.
Check out Linode or SoftLayer.
Is that really on the same direction that they want to serve startups?
By which time the phisher has already done the damage and moved on.
Campaignmonitor asked us what was going on - while this is debatable they may have been damaged from the complaing
But even linode asked us about it, just because the website where people signed up was on a linode hosting.
1 complain, nothing to do with hosting a part that the link is the here, and they asked us if there was something wrong.
I wonder if you can just blast a spam email with a competitor website (hosted on linode) inside and get him offline...
Then again, no one ever got fired for switching to Linode.
Is it just me, or are their other datacenters slower than their London one? It might be the latency, as I was actually in London, but the one I have in Georgia doesn't seem to have that fast disk accesses... Bottom line, though, the London datacenter is just time-traveling fast.