Hacker News Reader

Comment by viraptor | Hacker News Reader

Joining the chorus of other people saying "WRONG".

Reason 1: I pay for my bank account one way or another. It's bank's responsibility to keep the system running and secure, not mine. Sometimes we just can't do everything ourselves and have to rely on others.

Reason 2: Many of us host something somewhere. How many do backups? How many of us check the backups? How many do check the backups checking process? (enter infinite recursion) You have to stop at some stage. You probably don't have enough time, or your time is not worth enough to check that.

He can recover most of his blog from the caches, because it was quite popular. I bet he'll be back with an almost complete archive in less than a week.

View on HN

You have the perfect right to just point and say "you were supposed to do that", while losing all of your data.

The rest of us will be quite happy to ensure that what we think is happening, actually is.

It's akin to a pedestrian getting hit by a car and then saying "I had the right of way". Yeah, maybe so, but now you're in the hospital breathing through a tube.

Those of us who checked for traffic before crossing the street are at home watching TV.

The bank analogy is interesting and a good point, but the reality of ISPs is that they are less reliable, and less regulated, than banks.

If my income is dependent on data, I make sure it gets backed up. If it's an active project, I use my backups to build my dev environments so it's fairly obvious when a backup has failed.

You have good habits, and that's sincerely commendable.

However, it really isn't fair to blame this kind of problem on the end user.

Are website developers also expected to keep the servers secure? If Apache isn't patched and up to date, is that the website admin's fault?

Service providers are paid to do a job. This one failed tremendously, and should lose a large chunk of business for it. I think Atwood is being far too kind in accepting 50% of the responsibility for the data loss.

To put it another way: the only reason to maintain your own backups of your site data -- aside from healthy paranoia -- is because you expect your service provider to fail at doing their job. And if that's the case, shouldn't you be finding a service provider that does it better?

To put it another way: the only reason to maintain your own backups of your site data -- aside from healthy paranoia -- is because you expect your service provider to fail at doing their job. And if that's the case, shouldn't you be finding a service provider that does it better?

If you don't expect your service provider to fail, you don't know anything about service providers.

Everybody fails.

Our colocation facility has redundant generator systems. They're tested regularly, and have handled failures previously. Yet, when the power went out, three of the backup generators failed, and our site (as well as Craigslist, Yelp, and others) was out for 45 minutes.

The cause? A bug in the backup generator's software: http://365main.com/status_update.html

Shit happens. Sometimes it's not your fault. You still need to prepare for it.

>Are website developers also expected to keep the servers secure? If Apache isn't patched and up to date, is that the website admin's fault?

Getting hacked isn't as potentially catastrophic as not having backups. With backups, being hacked can be recovered from and the service provider changed.

>To put it another way: the only reason to maintain your own backups of your site data -- aside from healthy paranoia -- is because you expect your service provider to fail at doing their job.

A business doesn't expect its premises to burn down, but most have fire insurance in the event that this happens. Even if I don't expect my service provider to fail, there's no way to know that they won't and it makes sense to deal with this risk if the cost of dealing with it is reasonable and the cost of not dealing with it is catastrophic.

> It's bank's responsibility to keep the system running and secure, not mine.

As others have pointed out, it is your responsibility to check your bank statements to ensure that all is secure. But also, with banks you're dealing with money that is easily replaceable. If your card is compromised that money is gone but if it's their fault (or criminal) the bank will replace that money. Your data is not replaceable -- when it's gone, it's gone.

"Many of us host something somewhere. How many do backups? How many of us check the backups? How many do check the backups checking process?"

True you do have to stop somewhere -- but there is due diligence and there's negligence. Making sure you are making offsite backups and periodically testing them is due diligence. On the otherhand, completely relying on someone else to backup your critical data is negligence. If your data is at all important to you, then you need to have a copy. But at some point you have to accept that you've done enough.

You are responsible for reading your statement and ensuring that all activity is valid, much in the same way that you are responsible for ensuring the viability of your recovery strategy.

Reason 2: Many of us host something somewhere. How many do backups?

Anyone who wants to keep their data badly enough to pay (time, money) to do so. Coding Horror is a popular technology blog, and there's a significant cost in traffic and credibility when it fails.

Jeff Atwood lacked cognizance of the risks and made an incredibly poor technical and business decision in failing to validate correctness of his backups and implement a suitable recovery strategy.

You cut a lot from reason 2. The was an important part. I've seen a system where the backups were made. They were verified too. Only after an actual data loss it was discovered that the backup verification was faulty and half of the "verified" and "properly backed up" data was missing.

My point is that you can spend lots of hours trying to backup your data and verify its correctness. But unless you put it back into the actual working environment and check every single bit of it, you cannot be sure it was a proper backup - not with 1GB of information and certainly not with 1TB. And then after you verified that you can verify that you have backups, something will fail and a bunch of people will say "if you didn't check the backups properly, it's your fault". I've seen backup systems fail in amazing ways and will probably never again believe that you can be "sure".

It's interesting that you should mention banks. Banks routinely go under, and in almost all cases your deposits are only 'safe' up to a certain amount. Most data stores are worth well in excess of that amount, so if you could not reasonably expect a bank to hold on to a very large amount of data it stands to reason that you should not trust some service provider with data that has serious value without a secondary system in place.

After all, their liability will almost certainly be less than the value of your data, in which case there are two reasons to keep extra copies, both business continuity reasons and direct economic ones.

Banks make a bad analogy.

Sure, it should be the bank's responsibility to keep accounts safe. However, the fact that bank accounts are considered to be "safe" is in a large part due to systemic factors like the existence of the Fed (as lender of the last resort), FDIC, general political climate of "too big too fail", etc. As mentioned above, banks actually do go under a lot; it's just not as noticeable to the clients as before [the central banks], but the cost is there. (And there's a whole discourse there of whether it's a good idea for monetary system to have this kind of environment in the long run.)

In any case, nothing of the sort applies to the hosting providers (or even can be applied, as insuring unique data is not the same as insuring amounts of money) -- so assuming that keeping data there is as safe as keeping money in the bank is not a viable comparison.

1. Banks are more reliable than hosting providers. 2. For a small personal site you don't need anything fancy, just make sure you've SFTP'd the files to your disk and snapshotted the DB once every couple months. My personal site is backed up by Rackspace cloud, but I still do an occasional rsync just in case they fuck up.

I think there analogies for both sides of the argument. In the case of bank accounts (or bank security) I don't test it by trying to break in, check how it works with id theft etc ;)

But for backups, at least in this (early) era of manage hosting/cloud, its probably not too onerous to test your plans etc...

Reply on news.ycombinator.com