Pinboard.in service limited - FBI raided hosting company and pulled equpiment | Hacker News Reader

Pinboard.in service limited - FBI raided hosting company and pulled equpiment

status.pinboard.in

265 pointsrograndom81 comments

I love how Pinboard deals with this issue: site remains online (albeit with limited abilities), all data is secured and backed up, users are encouraged to use the export tool if they feel the need and there are status updates on Twitter.

Keep it up Pinboard!

One thing that does work is our export page, and if you're feeling nervous about your data I encourage you to use it. We have fresh backups (from one day ago) safely stored on S3, but in these matters you can't be too careful.

+1 Maciej. I am glad I paid for an account.

This is why I love Pinboard. They seem like they always do it right by the customer.

Not to mention it's dead-simple to use.

Dropbox's Arash should learn something from this.

We're not down. Our main DB server is unreachable and there is limited service (no API, search or feeds) while we run off a smaller backup server.

See http://status.pinboard.in/ or @pinboard on Twitter for updates.

Thanks, one question though... I've just been making some more bookmarks... is your backup server database now your master? So when you get your main databases back (or a replacement), will my bookmarks remain when you switch back?

It's a master-master setup with only one accepting writes at any time. So it should catch up first, and then I'll be able to switch back over to the original DB without data loss.

Good architecture. And nice of you to explain clearly and completely the problem and to invite your users to backup. I'd like to see that more often, but more companies are too scared to lose customers to do that.

But they'll win new ones when this story and the likes come to their attention, and when they learn how the company handled it with honesty and aplomb. I for one was unaware of this service and this story has made me consider signing-up.

@idlewords: would you have stats at hand?

Maybe a temporary loss of functionality earned you a few users above the daily average.

Which is exactly why I'm staying with pinboard!

I am the founder and current CEO of a dedicated server host. The FBI and other law enforcement agencies do often contact us regarding activity emanating from or related to our network. Its the same for any host and the bigger you are, the more often it happens. Rarely, a local law enforcement officer from within or outside of the US get cranky when you tell ask them to get a subpoena (makes work for them). However, in my experience the FBI and other three letter agencies are very professional, fair, and genuinely want to do it right. Getting a subpoena is no big deal at all for them. Usually the US Attorney is right down the hall from them, in the same building.

They do officially have policies and programs designed to befriend local businesses in the area of a field office. They just want to catch the bad guys and not mess with the innocent. They want people to like them and trust them. They count on the cooperation of businesses, especially hosting and access providers. There are civilized processes for everyone to get what they want.

There are the cases of people trying to get information who are not actually law enforcement, and that is one of the many reasons you must ask for a subpoena... To protect yourself and your customers privacy. The FBI can get one in a very short time.

Where this all goes bad is when you do not respond to subpoenas for subscriber information, when you don't hand over the disks, etc. If you do not comply, then what alternative does the FBI have but to come and get it?

My guess?... The host didn't play nice with the FBI.

Or the FBI believed the host was aware of, sanctioning, in cahoots with, or participating in the alleged activity and thus could not tip their cards.

This happens more often than you might think.

The FBI is just another in the list of bad things that can happen to your servers. Flood, fire, theft, law-enforcement... its all the same thing to a hosted service.

Off site backups, executed masterfully here saves the day.

There are a few services that I run on servers in separate countries. Failure of reasonable rule of law within a single country is a failure mode we consider.

Sadly this isn't implied, so let me suggest: "Off site backups with a tested restore process saves the day"

This was a failover, not a restore. And we were just lucky that the failover machines (in the same datacenter) were not affected.

Do you think you will change hosts and/or have secondary services elsewhere?

Why not spin up some EC2 instances if your main datacenter fails? You can pay minimal costs to send a diff of your database a couple times a day in order to keep the backup current, and only pay for real bandwidth if your main server goes down. Seems to be a pretty decent backup plan.

Is it possible to insure / sue for careless seizures by government agencies? Pinboard is (probably?) not the one they are after. What recourse do they have?

I believe the current status is: no definitive resolution of whether there should be any compensation required, but, paying compensation is not typically done currently, and no strong precedent requires it, while some precedent holds that it isn't required.

A 2008 post by a libertarian law professor (http://volokh.com/posts/1209706276.shtml) argued that the Takings Clause (which requires compensation if e.g. land is taken to build a highway or military base) should also require the government to pay compensation to innocent third parties inconvenienced or harmed during the course of a criminal investigation. But, that post noted that no high courts had actually held that, and at least one appeals court had just held to the contrary.

This is in the normal case, at least; things would probably be different if it were deemed to be some sort of overt act, e.g. if a police chief orders a raid designed to intentionally damage an innocent third party's interests (and you have evidence to show that).

It does seem that these ham-handed seizures are reaching constitutional areas. "Unreasonable" seems an understatement.

What would happen if the police shut down entire city blocks because one business in that block was breaking the law?

Even if it were, consider the costs of doing so. You'd have to do a lot of due diligence to acquire the insurance and prove to the insurer you're not high-risk. Then there's the premiums to pay for the policy, plus your legal fees and engineering costs to attempt to prove to your insurance company that an incident was the result of carelessness by law enforcement so that you could collect on your policy. Not to mention the opportunity cost (both short- and long-term) you're hit with from the downtime as well as from all the employee time that is sucked up in an attempt to seek restitution and eventually bring your business back online once things get sorted out.

All those costs would dwarf the cost of building a fault tolerant architecture from the start. Having a distributed architecture protects you from random law enforcement activities, power/communications outages, or even just glitchy hardware. Not only that, but it offers the advantage of built-in scalability should your business exceed your current capacity. Unless you have massive amounts of legacy infrastructure, data, and code which makes it cost prohibitive to migrate, it's really cheap these days to have a proper distributed architecture from the start.

If you can apply the concept of hot pursuit here, they exceeded their reasonable expectations. It'd be the equivalent to running though a block of homes to chase a man through one home. That said even applying hot pursuit is silly, but illustrates how this can be viewed.

The brave new world of overzealous law enforcement data warehouse searches just continues to wreak havoc on "cloud" services and content. It's akin to digging up the neighborhood to search somebody's house. What can the Feds/providers do differently to prevent this from happening so often?

It seems ridiculous that this is affecting so many customers - pinboard, Curbed, one Instapaper eval/test server, bunch of individuals - when there was one FBI warrant.

Apparently the FBI took three racks. If those are whole racks, it seems excessive. https://twitter.com/#!/Pinboard/status/83256217174147072

But another issue is that Digital One told Pinboard this was due to bad router firmware -- how do you confuse a software issue and an FBI raid? https://twitter.com/#!/Pinboard/status/83257679023325186

Most likely, DigitalOne doesn't have support staff sleeping in hammocks at the colo. If you were an admin, logged in remotely (from switzerland, say, where DigitalOne is headquartered), and you saw a few racks of equipment go offline, would you:

  1. suspect that someone had walked away with 3 racks, or

  2. guess that some network hardware had gone bad?

Usually it's a horse, but sometimes it's a zebra.

When there's an FBI goon breathing down your neck telling you to keep your mouth shut, and you're pressured by customers to say something, your options aren't real attractive.

Maybe the router issue asserted itself when they removed the machines they had a warrant for and took out the pinboard et al servers.

These are blade servers so you can fit a whole bunch of them into one chassis. Still trying to figure out why they told us initially it was router trouble, and what exactly the sequence of events was.

> What can the Feds/providers do differently to prevent this from happening so often?

In the United States we might start considering computer equipment as private spaces, not unlike our residents and vehicles. With so much of our lives increasingly digital in nature, it's not that screwy of an idea that law enforcement should require a search warrant for computer equipment and storage media.

    > In the United States we might start considering computer 
    > equipment as private spaces, not unlike our residents and 
    > vehicles. With so much of our lives increasingly digital 
    > in nature, it's not that screwy of an idea that law 
    > enforcement should require a search warrant for computer 
    > equipment and storage media.

That's not at all the issue. The 4th amendment has been extended to digital media:

http://www.cybercrime.gov/ssmanual/index.html

I'd be willing to bet my house that the law enforcement had a search warrant to take the data in question.

The problem is that the government's strategy -- grab the hardware -- doesn't work well when the hard drive is a virtual disk that sits on a server with a ton of other virtual disks.

Ultimately though, I think this isn't a question that can be solved by technical means. The solution is going to come when law enforcement can come up with a way of "confiscating" the virtual disks that doesn't open the door for claims of tampering & evidence planting.

I should have made myself more clear; I was aware that digital media is covered by the 4th, but failed to indicate this or discuss the following:

> The problem is that the government's strategy -- grab the hardware -- doesn't work well when the hard drive is a virtual disk that sits on a server with a ton of other virtual disks.

A fine point. I assert that the government should, then, have to receive a search warrant for each individual's data on the specific piece of hardware. Consider the search of an apartment complex: getting a warrant for one residence of the building does not grant search rights for every other residence. My contention is that physical disks with many virtual disks should be treated as multiple tenant property, as indeed it is.

Thank you for the comment and the citation.

The metaphor falls apart a bit: the cops can search 4B for drugs without bothering me in 2A, but when everybody's data is on one disk...

I would, however, imagine that the police would need a warrant to search any other virtual disk.

But the seizure of those disks raises an interesting legal question. Any lawyers know the answer?

Law enforcement does not require a search warrant, in your hypothetical, for 2A precicesly because the search of 4B leaves the residence of 2A unmolested. I do not believe the metaphor falls apart, at all: because the virtual 'tenant' vhd0 of physical hard-drive hd0 cannot be undisturbed by the search and seizure of the data in virtual 'tenant' vhd1--and might reasonably be considered to have the data subjected search and by definition seizure--law enforcement _should_ require a warrant for vhd0 and vhd1.

Going back to your hypothetical, if the resident of 3A were stuffing materials of interest into the crawlspace between their ceiling the the floor of 4B and said crawlspace could only be accessed through the floor of 4B, the interested authorities would need either the express permission of the residents of 4B for entry into their residence or a warrant for search. _That_, I assert, is a similar situation akin to the topic under discussion.

In the case of Pinboard, at least, they had their own machine(s), i.e dedicated servers not virtual servers or shared hosting. So at least some machines were taken the belonged entirely to people not subject to the warrant - it seems like the idea was just to take the whole rack, regardless of who else was on it.

In a sane environment, you'd be right.

But the government is going to claim that its "right" to get at the evidence is stronger than the property rights of the other co-resident users.

Consider: the 4th Amendment isn't in effect when you're within 100 miles (?) of the border; the chance of smuggling, etc., is greater there, so the need to detain and search people in that zone outweighs our rights. Or, consider a case in which my buddy borrows my car, and is seen driving through an unsavory neighborhood, is stopped, and found with a big sum of cash. This gives them the "right" to confiscate my car because it was involved in drug trafficking, even though no crime has been committed, and I wasn't involved in any case.

My understanding is that the CBP and immigration officers can work anywhere within 100 miles of the border ... or of any port. Which includes airports and general aviation fields capable of receiving flights from outside the USA. Which means just about anywhere.

However, that particular issue is not germane to the case in point, which appears to involve the FBI rather than immigration/border patrol.

Those constitution free zones only apply to stops and searches of people, not searches of property.

Your post is high on hyperbole and fear but low on content.

I disagree: http://www.computerworld.com/s/article/9177438/Border_search...

Wow, I'd better avoid all the hosting companies that repeatedly fly my data through TSA!

Just because the government has the legitimate right to conduct warrantless searches under conditions {x,y,z} doesn't mean they have rights to conduct warrantless searches under any other condition, or under all conditions.

>What can the Feds/providers do differently to prevent this from happening so often?

The feds could do plenty, but won't. The legal system is slow to react, and the feds likely fear that just taking virtual disks would lead to evidentiary issues at trial.

Hosting providers might be able to provision in a way that the servers hosting evil.site could be taken by the feds but backups existed of the other sites, but it would be an extremely hard -- if not impossible -- task.

In the end, cloud computing doesn't change the rules of safely deploying your site to the world -- if you care about uptime & your data, your site must be running on at least two different hosts.

> and the feds likely fear that just taking virtual disks would lead to evidentiary issues at trial.

Honest question: why are physical hard disks more valid as evidence than virtual ones? Aren't they equally easy to tamper with?

Well, sure.

But the police have a well designed chain of custody system to at least reduce the possibility of tampering. Not everybody in the office can tamper with evidence behind lock and key with seals -- of course it still happens, but the goal is to reduce the possibility.

This falls apart when the evidence in question is data that can be copied, altered, etc.

And of course we can likely come up with clever cryptographic answers to this problem, but in the end, it's far easier and a more understood procedure for the police just to grab hardware and seal it up.

Keep in mind that ultimately the chain of custody must be explained to a jury, to convince them that what they're being shown actually comes from the defendant's computer, while a defense attorney does his best to sow doubt in the jury's minds.

Do you want to explain to 12 randomly selected people how virtual disks work, and the cryptographic algorithms you used to ensure that the data you're showing them is identical to the data the defendant had on their system? Could you explain it so well that a reasonably skilled defense attorney couldn't confuse them enough to produce reasonable doubt?

You probably can't take a virtual disk; you could copy it, which answers your question: a copy is considered less valid than the original.

I don't think it does answer the question, though. The data is the evidence rather than the medium it exists on. What guarantee is made that the disk wasn't tampered with that cannot be made by a cryptographic signature...

...or is the answer that the law has no adequate treatment of abstract evidence?

Two different hosts separated by a distance.

Two different hosts separated by a border would be better.

Letting people that are familiar with what they're investigating to actually run the investigations would be a fairly decent start. In my experience with federal law enforcement, the support personnel and a few agents are usually quite knowledgeable and on top of things. The supervisory agents, however, are promoted into the positions based on their performance in completely unrelated types of investigations.

There may be safety in numbers. One can imagine that if the FBI showed up at an Amazon data center and was told "it's somewhere in these thousand racks" they'd rethink the "seize it all" approach.

In 2009, the FBI seized an entire data centre in Dallas. Not a server. Not a rack. Not a cage. [1]

Everything.

Moving rack after rack into waiting trucks makes for excellent TV.

Somewhere in the FBI, right now, is a man or woman powerful enough and ignorant enough to order such a seizure. It's pretty much inevitable.

[1] The original link is broken, but I discussed it at the time here: http://clubtroppo.com.au/2009/04/05/on-a-risk-i-had-not-cons...

I wouldn't bet any money on them changing their approach unless an Amazon lawyer is present.

In the time it would take them to load a thousand racks of equipment into the truck, I suspect Amazon's lawyers would indeed spring into action.

Unless they threatened the local staff to keep quiet (or whipped out the 'National Security Theater' card). I'll bet in either case, they would get smacked down hard by Amazon's lawyers though.

probably, but the FBI could also round up all the data center employees for questioning.

It's interesting to think through what would happen with AWS in the same situation.

Would the FBI turn up and say "Where's the 100 servers for customer X", then seize up to 100 different physical servers, depending on the distribution? Or even 100 racks worth of physical servers...

I think they just asked for a server on an IP and the datacenter staff (!= not the DigitalOne staff) said the IP was in the range of DigitalOne - so the FBI took them all. I think the datacenter will rent IP ranges to their clients.

I can imagine in an Amazone datacenter they would be able to point to the exact server they're looking for.

I think the OP's point is that Amazon EC2 can run 100s of instances for one customer which could be on up to 100 different physical servers and when not using EBS roots, any one of those servers might still contain the evidence the FBI wants.

Mentioning the specifics of Amazon further, what about if your account for the highly illegal operation was using/paying for EBS/S3/SimpleDB/RDS/SQS/SNS/SES (or all at the same time). Any could contain forensic evidence of a crime as they hold data in some sense. EBS especially is likely to run on some kind of SAN; would they have to crack open the racks and take out individual drives? would they have to take whole arrays of discs because of RAID-esque striping?

It's like a LEO denial of service (both on amazon and the forensic analysis). Pragmatically, they might trust amazon enough to consolidate/quarantine the data into the smallest surface area first.

What I find awesome is that instead of being disgruntled or disappointed, the way in which pinboard has handled this situation has reinforced my confidence and appreciation for it's service. They are truly a case study in customer awareness and communication for the tech industry.

Can a law-knowledgeable individual shed some light on what this means for individuals who weren't a part of the warrant?

i.e., I'm hosted on the same server as Joe, who the FBI are investigating. They seize the equipment we share. During the course of their investigation, they naturally also examine my data.

Do I then have a viable lawsuit or claim towards unlawful search & seizure, or invasion of privacy? Or, if I happened to also have illegal content on the same system, would they be able to use the evidence they encountered there in a case against me?

In general, law enforcement has to get very specific warrants accurately indicating where they're going to search and what they hope to seize. This includes both physical locations and, usually, computers. Additionally, case law makes a distinction between property that you personally own and/or control (e.g. your house that you live in), and property that is shared (e.g. the apartment you rent with a buddy), and recognizes that the part of something that is "yours" (your room in your shared apartment, for instance) is separate from you buddy's (their room). (I imagine this is because the 4th amendment protects your papers and effects from search and seizure, not just your real estate.)

So, all things being equal, the FBI is supposed to take pains not to even look at the data of people not named on the warrant (automated processes have been exempted from this, IIRC, as they aren't actual people), even though your data is housed on the save server as the person(s) named on the warrant. And if they did by chance see something they weren't supposed to, it wouldn't be admissible in court (in a case against you - they might be able to use it against someone else).

IANAL, I just took a couple of law courses in college.

> ...they might be able to use it against someone else...

That's an interesting point. So, if I'm running a hosting company, and they come across data from one of my customers...?

Also, putting aside for a moment IT best practices and all that, they potentially are crippling my business by seizing machines which I share with someone else. Is there any recourse for that?

An article on HN a few months back about a guy who worked in a Three Letter Agency specifically stated that when or if a data analyst came across data that was not pertinent to their job, and viewing that data was potentially infringing on someone's constitutional rights, there were forms filled out and steps taken that stated such an occurrence happened. He also said the Patriot Act screwed that all up.

The WP article below specifically talks about communications, but the skeptic in me doubts there is much change between one form of data and another in a highly bureaucratic environment such as the NSA.

http://en.wikipedia.org/wiki/Thomas_Andrews_Drake

http://www.washingtonpost.com/wp-dyn/content/article/2010/07...

Thank you.

I just started using Pinboard a couple days ago (mostly because of integration with Instapaper), and so far it looks great. The fact that, even with all this chaos going on, the main functionality of the service is still up and running is really great.

Really impressed by your ability to weather this storm. Definitely going to take a closer look at the service and see if it's useful to me (I'm a kind of 'write-only' Instapaper user... could Pinboard help improve my read rate at all?)

I've been a Pinboard customer for a long time, I think it is an example of a well executed, very well designed service that just blends into the background even when I use it a dozen times a day.

I wonder if the law enforcement folks look at the "collateral damage", i.e., all the innocent servers (virtual or otherwise), as a windfall of extra data to mine. If my neighbor gets his house raided by the SWAT team they do not have access to my house only his, but with hosting this is not necessarily true. Further, I at least know who my neighbors are and can choose to not live in a bad neighborhood or even move away if I want, however with hosting this is not the case.

Suddenly, the PCI compliance requirement to encrypt all sensitive "data at rest" makes a lot more sense. Probably not the case with Pinboard, but interesting nontheless.

Can't wait to read the write-up of that.

A site I visit frequently got its server raided by the FBI (the host was rewiredHost; it was unrelated to the site I visit). The site had to start from scratch except for their frontend code.

I wonder why the FBI is raiding all these servers. Another comment mentioned a few hosts they've hit.

A popular method used by hackers is to sign up for a virtual server with a stolen credit card. If they are careful and only access it through a proxy, their hacking attempts are virtually untraceable. With the amount of hacking going on lately by Lulzsec and other groups, there is bound to be a lot of collateral damage.

Sounds like a very black hat way of taking out your unprepared competition. I'm sure you can buy this sort of service somewhere - get a hit on a datacentre/rack by security services in response to highly illegal activity perpetrated in the open in order to cause the most possible disruption.

I didn't mean to imply they would intentionally do that to take out competition, although that is indeed a possibility. The main purpose of having a stolen virtual server would be to launch attacks from.

The truth is even if you rented a virtual server in the same data center, your chances of being in the same rack of hardware are pretty slim.

If the data was replicated in N places would they serve all N places? Just curious how that works.

It seems civilian causalities will always exist.

Here's where hosting on EC2 might actually be worth it.

Let this be a lesson to chose your hosting company wisely. You can't always predict such things, but I think if you pick a big name, with a good business reputation, you'll minimize the risk. They might do checks for abuse and have plans in place to keep everything up when something goes wrong.

Eg. I can't imagine that if you host with http://www.rackspace.com/ your website would be down if you didn't do anything illegal.

DigitalOne's site is also down http://www.digitalone.com/ so they don't even have a backup server for their own website..

http://www.eweek.com/c/a/Cloud-Computing/Final-Thoughts-on-t...

Shit happens.

All big names were once small. I doubt pinboard didn't choose their hosting company wisely, although I get what you're saying. Troubles and down times happen to anyone, it's how good you can deal with them that makes a difference, not the hosting company you choose.

Ofcourse, there's no blame on pinboard, they handled it amazing! And the webhosting company was around for 7 years if I googled it correctly - but I do think there's a big difference between webhosting company's.

Suppose DigitalOne rents space in a datacenter and has no one on-site (exept for staff from the datacenter). I can imagine that if the FBI enters there "we need servers from IP x.x.x.x and y.y.y.y immediately - that the people from the datacenter just point to the servers from DigitalOne "oh that's in their range"..

Ofcourse that's just speculation if you don't know how it really went down - never the less, I would chose webhosting carefully if your business depends on it:)

Reply on news.ycombinator.com