F.B.I. Seizes Web Servers, Knocking Sites Offline
bits.blogs.nytimes.com
bits.blogs.nytimes.com
* Make sure you failover to servers in a separate cabinet.
* Better yet, make sure you failover to servers in a different data center.
* Best Possible – Configure a task to switch your primary and
failover environment from data center to data center
once a week at random times. Or eliminate the concept of primary
and secondary altogether.
The only solution is backup hosting in a separate data center.
Please, that doesn't make FBI seizures any more palatable.
Personally, if I heard the FBI raided a datacenter and knocked a bunch of servers offline which had nothing to do with what they were after, then I'd seriously question the security, legal team, and response team of that place. To me, it sounds like the FBI just showed up to one guy sitting in a chair watching TV and let the FBI roam free.
"The raid happened at 1:15 a.m. at a hosting facility in Reston, Va., used by DigitalOne, which is based in Switzerland, the company said... DigitalOne had no employees on-site when the raid took place. The data center operator, from which DigitalOne leases space, passed along the information about the raid three hours after it started with the name of the agent and a phone number to call."
Any data center that has been around for bit will have random screwups to a tiny percentage of their infrastructure almost on a daily basis.
So the next time the FBI decides to knock over someone's rack, the impact could be a lot wider than a handful of sites.
US security agencies are becoming increasingly cavalier when it comes to seizing domains and hardware. This is becoming a significant risk for online startups like my company and the companies of other HN members. The suggestions here to mitigate this risk are not cheap. Setting up a fast enough link between data centers to have real-time replication is prohibitively expensive for most small companies.
I'd like to see legislation that lets us know what our rights are and that lays out a standard procedure for these kinds of data center incursions. I'd hate to see a cloud provider's rack get taken down in one of these raids, and I don't think any of us are happy about the government using our tax dollars to settle costly lawsuits caused by their own incompetence.
It's like a meteor strike: well reported, flashy, but very very not likely to actually happen to your site. How many websites do HNers collectively operate? How many have ever gone down because the FBI took their hardware? How many have ever gone down because, oh, the hard drive crashed? Because they pushed bad code live? Because they misconfigured a firewall? Because the hosting company had a network or power issue? Those are real risks for your business. (Bonus points: many interventions for these and similar issues fixes your FBI problem, too!)
In the event of a totally freak incident like this, you're probably going to have downtime and a day's worth of data loss, but recovery for most folks here is likely "Reimage the VPS(s) from the latest backup, hit the on switch, and change DNS records."
Yes, if you look at the risk of a US government raid on servers, domains, or what-have-you in the past 5-10 years, you can fairly compare it to a meteor strike or a "freak incident". But things are changing, incidents and "collateral" are increasing, and if you look at just the past half year, you'll find it's gone from a meteor-strike level probability to a very small but definitely not dismissable chance.
"...and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."
When they seize unrelated hardware they are clearly overstepping their bounds. I'd like to see the innocent parties take this to court and put a stop to this. We are far past the time when the FBI can claim ignorance as to how a colocation site operates.
They do officially have policies and programs designed to befriend local businesses in the area of a field office. They just want to catch the bad guys and not mess with the innocent. They want people to like them and trust them. They count on the cooperation of businesses, especially hosting and access providers. There are civilized processes for everyone to get what they want.
There are the cases of people trying to get information who are not actually law enforcement, and that is one of the many reasons you must ask for a subpoena... To protect yourself and your customers privacy. The FBI can get one in a very short time.
Where this all goes bad is when you do not respond to subpoenas for subscriber information, when you don't hand over the disks, etc. If you do not comply, then what alternative does the FBI have but to come and get it?
My guess?... The host didn't play nice with or respond promptly to the FBI.
Fun times.
If the warrant is valid, I don’t see how anyone affected by the raid has a case. If the police serve a warrant on the house next door to you, and your street gets blocked off while they go over the building with a fine-toothed comb and carry off evidence, it will be damn inconvenient for you, but I don’t know of any law preventing them from doing it.
Oh yeah, and they're searching through your store as they make their way to the location on the warrant.
Good luck!
In the case of a shared server, I don’t see how you can do a good forensic search of the affected server without having the physical box in your possession.
One wonders whether their actual goal is not to gather evidence, but to actually interrupt the service.
There's a pretty large range between Pinboard (10 000 customers, maybe? Two employees? Annual revenue in the range of $100k? I'm guessing here) and Rite Aid (109000 "associates", maybe 20 million customers, $26 billion revenue). More than five orders of magnitude, actually.
But you may also read it like that if you like.
Where's the downside to this? The unrelated sites suffer maybe a few hours downtime max, and they don't have to worry about tipping anyone off.
If the FBI is already there, it really shouldn't be a problem for them to locate the exact server that hosts their alleged offender and confiscate only that one.
How do you think anyone would be tipped off in that situation?
For example, I believe it's illegal to take information which was coincidentally seized along with legitimate evidence subject to a warrant, and use it in an unrelated case. I'm strongly in favor of such laws, to discourage "fishing expeditions", where law enforcement uses a legitimate warrant to seize a bunch of unrelated material that they're interested in using for other purposes.
However, I suspect if you walk into a data center where some malicious customer is doing something illegal, probably that customer has tried to make it harder to connect them to what they're doing.
Also, I don't know about this case, but there are lots of small hosting companies that lease servers from other companies, and the staff at the colo only know the lessor, not the lessee. They wouldn't have any access to the hosting company's customer database which might map customers to servers.
Besides, the FBI has to worry about low-probability cases like, what if one of the employees is a friend of, or paid off by, the bad guys? Or what if the bad guys are somehow monitoring the facility?
The FBI has a legitimate goal of seizing the evidence they need as quickly and with as little notice to the bad guys as legally possible.
Did the FBI act wrongly in this case? I don't know enough to tell.
That is exactly what they are worried about happening.
There's a long way to go from having enough bandwidth to serve your personal blog or small-time web app out of your {apartment, house, office} to getting what you'd need for a large-scale web app (at a minimum, redundant 100 Mbps fiber connections with five-nines SLAs) routed there. That and getting a nominal 100 Mbps connection from your friendly neighborhood telco is a lot different from getting a 100 Mbps connection you're expecting to saturate 24/7. The telco won't put up with that for very long, because they overcommit their subscriber bandwidth in the (correct) expectation that most of their customers will not use all of it.
Deleted comment
That raises some interesting questions, the impression every seems to be operating on is that they stormed in and took 3 cabinets or full racks with computers owned by other companies. All things being equal, this sounds like the normal operating procedure for the FBI.
So these cracker groups have finally woken the sleeping giant? And the bleary-eyed half-awake giant is swinging his arms around wildly, demolishing everything in reach? Maybe after some coffee and donuts they'll be a bit more delicate.
If it's LulzSec, as the article assumes, I would hope they have good reasons to do so. LulzSec is a group of hackers searching for recognition. By chasing them they are giving them more publicity.
Host your site/content across multiple cloud providers. Then host VPS web servers with other providers and have them balance between the backend cloud servers.
Then put cloudflare on top of that.
Ideally, you would want to have multiple service instances with the cloud providers which are silo'd from each other - but exact duplicates (e.g. they are under fully different accounts with the cloud provider)
Your front-end web servers keep a cache of content from the cloud servers - and they are pushed to. For even lower-end content needs, you can pull from a drop box account or something.
Any of the front end gear gets seized - You can bring it up with a VPS image really quickly.
If you don't know what the word latency means, this setup will teach you all about it.
Another problem is that your various providers will probably charge you for WAN bandwidth. So you will pay three times for every request: Twice from backend to balancer (one charge from provider A, one from provider B) plus another charge to send the same data back out from the balancer to the customer.
Plus you will be miserable trying to keep your site up 100% of the time across two cloud providers. Have a problem on either one, and 50% of your capacity will go offline.
Might be better to realize that "my servers were seized by the FBI" is a rare occurrence and you can probably afford a few hours' worth of downtime and/or data loss. Make offsite backups from your primary provider on a relatively long timescale (once per day, maybe once per hour if you're more sensitive; live database replication for the crazy-sensitive) and have a procedure for spinning those up at a secondary provider. Test that procedure every month or so. The beautiful thing about cloud services is that you can pay for your emergency-backup servers by the hour and only when you are using them, or testing them.
If you're doing something where you might get the FBI seizing your servers - maybe latency is not a paramount concern.
Capacity might also not be of primary importance.
I am just trying to find the 100% most resilient form of online hosting that masks the layers as much as possible.
The tin-foil-hat in me can see many many reasons why one would want to be aware of how to accomplish something like this.