Instapaper Server Update | Hacker News Reader

Instapaper Server Update

blog.instapaper.com

163 pointspsychotik47 comments

I think you're more than justified in assuming no compromise happened. Someone with legitimate authority had physical access to your server. That has happened before and will happen again. Your server's cage opens several times a month, probably, as techs and whatnot do their icky atom-touching sorcery. Physical access to your server by the adversary means untraceable compromise, sure, but if that is true, hasn't it happened before? (Sure, the techs might be more trustworthy than the FBI... Unless they were the FBI, which is paranoid squared, but if you think that agents executing a warrant routinely grab servers in the vicinity because, hey, computers tend to have juicy stuff on them and we have way too much free time, then assuming infiltration seems almost reasonable by comparison.)

In a threat environment this bad, you would personally be scrutinizing sign-in lists every month. You have better things to do, right? Right. Because this is really not Instapaper's threat environment. You got mildly inconvenienced as a side-effect of routine government operations. You can pick whether you want to escalate that to A Full Crisis. That seems like a bad call, which you recognize on the legal side. Do it on the security side, too.

You may think that vocal parties will disagree with "This was a non-event for us.". They might. Screw them. You're a CEO; dealing with paranoid ramblings does not generate business value like two months of normal operations does.

Isn't there a way to get information from HDD's about how long they have operated from when they were manufactured using SMART? Not that anyone keeps those records on hand once the HDD is no longer in their possession, but if you are paranoid, you could start offloading those records every minute to a different server. That way if yours is seized, you could at least know if someone had turned on your HDD's and for how long.

It'd be cool if HDD's/SSD's started compiling rough stats about numbers of reads and writes for this purpose.

+1 to the attitude. It's all about getting the service back up and making it better. We (collectively) tend to spend too much time trying to figure out who to blame instead of just doing the cool shit that needs to be done. Kudos!

Sadly, I simply cannot agree. I absolutely understand Marco's desire to focus on developing Instapaper, but there's quite a bit more at stake here, and his willingness to move on without even a bit of protest is concerning.

He claims that there's nothing that can be done, but there are definitely legal actions one can take in this scenario. Sure, they might not ever fully compensate him, nor convince the skeptics, but it will at least reveal the details behind the seizure, and it will send a message to both the FBI and to others like DigitalOne that could prevent further "seize first and ask questions later" style operations.

So for a one-man software shop, given the choice between:

a) Spend money and time on legal action against the FBI for holding onto his servers for a couple of days while in the process of an investigation, in order to hopefully try and make a larger point.

and

b) Spending his time and no money by actually improving his bottom line by enhancing the features of his software,

you expect any rational person to choose (a) ?

Sorry, but the suggestion that fighting for justice is somehow the thing only "irrational people" would choose totally pisses me off.

I totally understand someone making a rational decision in favor of other priorities. It's what most people would do. But thank god for people who take the road less traveled. It's to them we owe the rights and freedom the rest of us take for granted. There's nothing "irrational" about that.

A major thrust of what I'm saying is in the fact that Marco is a single indie developer. Not a large company with more resources.

Fighting against the federal government over this seems like picking the wrong battle. He got is server back in a couple of days. Obviously its likely that the feds have already cloned it but its not like they kept it indefinitely or wiped it clean.

Does it suck? Yes. Is this something he should be fighting over? Not in my book.

The road less traveled is starting and running Instapaper. Chasing the FBI is a waste of time. If you really care about the problem, start a "company" meant to provoke the FBI into accidentally procuring its servers.

This would seem the perfect situation for the EFF. At some point someone needs to choose (a) or else it will keep happening.

Perhaps. But a one-man team probably shouldn't be the one choosing that option.

Depends on the market you're in. I'd say overall, Steve Jackson Games gained enough publicity to make it worth their lawsuit in a different illegal-search-and-seizure case, especially when you add the $300k they got ($500k in today's money) from winning it. They had a stronger case, though, since the raid was directly targeted at them, and fairly carelessly executed.

I was more severely affected than marco and I'm certainly going to pursue this matter to find out what happened. I imagine the Curbed team might share my curiosity.

We were significantly harder hit than either you or Marco, looks like. Our entire stack of 3 servers - app server and db master/slave - disappeared and remains offline to this point.

I'm curious about what happened, but ultimately I put this down to an unfortunate side effect of using blades.

Great to hear that! As a customer from pinboard I really appreciated your communication during this.

Agreed on this point. You have to choose your battles, and this is a lose-lose battle. Even if he would fight, win and receive some type of compensation, that's unlikely to be greater than what you had to give up in the fight.

I wish more people thought the way you do. You, sir, are an inspiration.

Tangent alert! Marco makes me feel great.

One of my favorite things about Marco is that he gives me confidence that there is not a gaping wide chasm between the skills of the best engineer-entrpreneurs and me. This time, it was his frank admission that he was taught by commenters and emailers than his current password encryption was unsafe and that there were better options out there.

When you realize that someone as successful as Marco has the same learning to do that "the rest of us" do, it makes me feel great.

Why do you assume that Marco is one of "the best engineer-entrepreneurs"?

News flash: Security breaches happen because almost everyone in the industry sucks at this stuff, including (most importantly) those who SHOULD KNOW BETTER, yourself and Mr. Arment included.

And now we do. :-)

Out of curiosity, does anyone know how volume-level encryption (like dm-crypt) holds up against this sort of thing? I find myself wondering what I'd tell customers if my server were seized as collateral damage like Instapaper's were. Will that sort of encryption serve as a plausible safeguard of customer data, or is it more of a padlock (easily broken with the right tools)?

It would certainly make it much harder to steal the data -- and at the same time it would make it much harder to boot the machine, because at some point you'd have to enter a passphrase. It's possible to do so over SSH, but it's still a bit tricky. Rebooting the server while you're away from an internet connexion will leave it in a useless state.

That being said, even with volume encryption, the key is in memory somewhere. RAM isn't wiped as quickly as you might think, and it's apparently possible to extract keys from memory after the machine has been powered down. So be warned, and be paranoid!

Indeed, the encryption keys can be recovered from memory fairly easily. http://www.usenix.org/events/sec08/tech/full_papers/halderma... is very readable; http://www.youtube.com/watch?v=JDaicPIgn9U is a well-made 5-minute introduction by the authors.

I thought this issue had been partially addressed by overwriting the memory containing the keys on unmount. Of course, that doesn't work if somebody just pulls the plug on the machine.

There was an article I read recently about a patch for the Linux kernel to store the key in CPU registers instead of memory as well...

A lot of servers I've seen have had 'chassis intrusion detection', which is a fancy way of saying they had a microswitch hooked up to one of the SMBus lines that could detect when the case is opened.

Take one of those, and a Google-style Lead-Acid internal UPS, and you'd have an incredibly hard time getting access to keys held solely in memory.

For the extra paranoid, accelerometers or a simple mercury vibration/tilt switch would make it even harder, at the risk of losing/hanging your server every time someone worked on the rack, or drove a heavy pallet cart nearby.

I recall seeing somewhere a device designed for law enforcement, which was essentially a cart-style UPS with probes designed to be clamped onto, and penetrate, the power cable, allowing seizures without any power-loss, which I assume is for this eventuality.

You don't necessarily need to encrypt the boot volume.

You tell them what Marco told them.

Assume all data on the drives has been cloned.

There's little chance they didn't.

He should absolutely assume that the data has been cloned, but there is about a 99% chance that it wasn't.

If the data was cloned on-site, then we wouldn't have heard about the search warrant (there'd have been a gag order).

And as this happened yesterday, there's almost no chance that the server could have been taken to the lab, imaged, and then returned to be reconnected today.

I would bet 100 dollars that the actual target server of the raid is still sitting on a dolly in the forensics cage while a tech is waiting for someone to tell him what he's supposed to be looking for.

You'd be surprised. I've worked with forensic law enforcement in the past, and they were equipped to clone drives quickly and easily. The idea is to create a clean image of the drive that they can use for evidence, and they don't boot the machine so if there's any trapdoors or whatever they won't hit them. And this was a local police force over 5 years ago, I can't imagine what the FBI would be capable of these days.

The FBI has the capacity to clone drives quickly, absolutely (although amusingly, the process is actually slower now than it was then, as drives are larger).

However, field cloning is absolutely not the desired procedure to use (for a lot of reasons). You only do field cloning if you are either: 1) Under a time crunch, or 2) Don't want the target to know that the clone has occurred.

If at all possible, you just take the equipment and process it in the lab.

I've done about 2 dozen of these for the FBI. We did field-cloning maybe three times.

Some should build a hard drive that logs being powered up and powered down!

I wouldn't be surprised if SSD's somehow had information like that stored in them (or at least the ability for it to be stored). Thankfully I got out of the Forensics racket before they were mainstream.

SMART power cycle count?

99%? I'd assume that the first thing they do on seizing a machine is pull the hard drive out and drop it in a disk cloning machine?

I like to always follow the saying "Hope for the best. Plan for the worst".

Assume the FBI has a copy of everything on the drives and assume that they will see every file on the drive, even though odds are they don't and they won't.

What do you base this upon?

He should contact the FBI and continue to beat on Digital One and determine if it was seized or not. It very well may have simply been turned off, is that not a valid possibility?

"I’m not convinced that they did everything they could to prevent the seizure of non-targeted servers, and their lack of proactive communication with the affected customers is beneath the level of service I expect from a host."

I can't speak to the level of service you did or did not receive but I seriously doubt DigitalOne could have done more to prevent "non-targeted" servers from being seized. If it was in the same rack as the "targeted" server I'm sure that 100% of the time LE would take it and find out whether it was related later. It's their job to take every step necessary to preserve evidence first and foremost when executing a search warrant like this.

I agree with you that the data was almost certainly not compromised and I think it's great that you're so proactive about protecting your customers by moving to bcrypt.

This is all just assumptions, since amazingly the hosting company still has no comments at all. Who exactly rebooted the server? Not the FBI, for damn sure.

I hope he securely erased the data rather than just deleting it, otherwise it might not be just the FBI who have potentially got a copy...

Would there not also be backups at DigitalOne?

With dedicated servers, the responsibility is on the client to back up what matters to him. Ditto with VPSs.

I think a neat idea would be to create 1U - 4U lock doors that could be bolted in front of a server in a rack.

I wonder if this would be enough to give each server it's own legal "separate space" insomuch as to prevent a whole rack from being seized due to warrants like this.

The server that was taken from Marco was a blade server, and he just rented a single node in the blade. Locking the single blade he was on wouldn't have prevented the situation that occurred, as the FBI seized a number of enclosures, rather than servers. Most likely, the agents who took the servers didn't understand a blade server, and took the entire enclosure, rather than pulling a couple nodes out of the blade.

Should services consider storing all user data on encrypted volumes?

Pro: after a powerdown, seizure/theft/cloning of the volume won' reveal user data

Con: need a secure way to supply decryption passphrase on each reboot

Man, you have an awesome service. It served me greatly through my post secondary education. I am glad you got your server back, and you can keep on keepin on.

Again, thank you for the services you offer.

This is the exact opposite approach dropbox takes - well done with good communication. Props.

Reply on news.ycombinator.com