Linux-Kernel Archive: RIP - dead harddisk | Hacker News Reader

Linux-Kernel Archive: RIP - dead harddisk

lkml.indiana.edu

41 pointsbsg7552 comments

I have Arq (http://www.haystacksoftware.com/arq/) set to back up my ~/git folder every hour. At most I can lose an hour of work.

This came in handy once when I accidentally did the equivalent of rm -rf ~/git. Lost 54 minutes of work.

Arq is Mac-only, but you could accomplish the same thing with https://www.tarsnap.com/ and a cron job. Both are changes-only backups so there shouldn't be a huge performance hit.

I also use Time Machine when I'm at home but only sort of trust it.

Has anyone got any experience with using Dropbox as a backup service? They provide 30 days of file recovery for free, and for a few dollars a month you can get packrat, which provides eternal point-in-time recovery. Seems like a great use case of a git folder backup?

I use it like that. iTunes, development, documents, all sorts of things. Easily synced to a new machine if needed.

On top of that I pay for Backblaze, just for a little bit more protection.

Dropbox is not a backup service, it's for synchronizing local folders.

I've had weird issues with my whole Dropbox getting deleted - in the middle of the night, with my machine off, have never figured out why - and while your Dropbox can be restored through the web interface, it's just one file at a time. In the case of a "mass delete" you actually need to e-mail support and generally wait a couple of days before they restore everything, which is less than ideal.

The last time it happened (the third time!) I wrote some scripts myself using their API to do the restore rather than wait on support. It really doesn't make for an ideal backup solution, though!

On Linux it is extremely easy do use rdiff-backup for local and remote incremental backups. When I work, I have a minutely cronjob for my work directory. For the whole system I have a hourly schedule. When I resume work the next day, the previous mess of diff files from the work rdiff target directory is deleted and a new clean "emergency" cycle is started. Losing one hour of great concentrated coding can be extremely annoying and hard to reproduce so this is a cheap way to prevent it. But I guess even an hourly backup would be a good first step for many people.

Remember, a harddisk probably costs less than one day of your work. Or just get an USB stick for the most important bits.

Count me as a another happy user of rdiff-backup. It creates an rdiff like copy of what it's pointed to and deltas of previous changes. Not quite continuous data protection but close enough.

I have a Intel 520 SSD. Fantastic thing I must say and I don't think I can ever use a HDD again, but somehow -- and I realize this may just be baseless -- I don't trust it as much as I trusted my old Seagate HDD, which lasted 7 years.

SSDs are getting better now, but are they as reliable as HDD? Can anyone shares his/her expertise on this?

You don't have a reliable option. It's that simple.

If you value your data, keep backups. And if you want to avoid hiccups in productivity, use RAID. Data that you don't have backed up is data that you don't have.

And keep in mind that RAID, while useful, is not a backup. It doesn't protect against theft, whole-computer destruction, human error, or many other things.

You can certainly buy SSDs now that are rated to more read/write cycles than your 7-year-old Seagate will have gone through. The ones we buy are rated for 3+ years of constant I/O at full bus speed. I don't know how much of that tech is in the sort of SSD you're likely to buy for a desktop, but they're certainly available.

> SSDs are getting better now, but are they as reliable as HDD? Can anyone shares his/her expertise on this?

I'm especially interested to hear from people who use many discs, and if they can share some information about the kind of use those discs get.

From what I've heard SSDs are more reliable that traditional spinning platter drives, unless you have a bad batch. This is especially true if there is more than one platter in the drive.

But I also recognise that it doesn't really matter. All drives fail, and no drive is reliable, so you assume the life is going to be 3 years and you have robust mirroring and backups and replacement policies.

> No drive is reliable, so you assume the life is going to be 3 years and you have robust mirroring and backups and replacement policies.

A thousand times this. Over the last 2,5 decades I lost far more data than I’d like to admit. Floppy disks and CD-Rs that went bad, hard disks that failed, FTP services that were discontinued or corrupted data, etc.

My backup strategy right now:

* I have SSDs for use as system disks on all my computers. I keep track of their use and how long I’ve had them. Once one of them turns 3 years old, I’ll swap it out for a new one.

* Those computers are all connected to a Time Capsule (Time Machine) for incremental changes.

* Some files are also continually saved on iCloud (not all apps support it.)

* At night, a real backup gets run from the computers to a Drobo RAID setup.

* Everything of value (files I created) gets sent to Backblaze daily (an online backup service).

* For web projects, I also use a hosted subversion service.

* Most media I buy is cataloged by iTunes Match, where I can download it again whenever I need it.

* I use Dropbox, but not for backup, I use it to view files on the go and to share them with others.

I know my setup is not perfect, and I could use advice to improve it. I haven’t lost data in the last 3 years, but I’m keeping my fingers crossed.

I do count myself lucky to still have all my emails dating back to 1998 (but not the emails from 1994-1997). In large part, I have iCloud (née iTools, .mac, MobileMe) to thank for that.

From what I've read online, SSD now should last the same, if not more than HDD.

The nice thing about them is that you know how many writes they should handle, and get a new one before some disaster happen. Sure, you can get a faulty unit, but you could get one with HDDs as well.

Instead, for HDDs, you couldn't be really sure. Yes, there's the "broken sector" count, but this isn't always reliable. Disaster could always happen randomly, without any warning before.

If you're interested in more stat, some sites are starting to run endurance tests [1][2][3][4], to see how much on average an SSD is supposed to last, and which brand is better.

Anyway, the only real solution is to run a raid, and keep frequent backups.

[1] http://techreport.com/review/24841/introducing-the-ssd-endur... / http://techreport.com/review/25320/the-ssd-endurance-experim...

[2] http://www.xtremesystems.org/forums/showthread.php?271063-SS...

[3] http://ssdendurancetest.com/

[4] "SSD Endurance test" on Google

My SSD died. Much like a USB drive, when it died, I could still read from it. Just made a direct ghost of it onto another SSD. Was up and running in less than 15 minutes.

It really depends. If you keep your PC running all the time and are always writing to it at pretty high bandwidth then you'll wear out a SSD really quickly and a HDD won't have the spinup/spindown cycles that impact it's lifetime. If you're not in that group than, on a mechanical level, a SSD will tend to last much longer.

On the other hand, the abstraction layer that lets a SSD be so reliable is complicated and it's possible for companies to screw up complicated things. There have been several instances of SSD firmware bugs that have caused data loss for users before they were fixed, so maybe be wary of new SSD models.

I've been using Intel SSDs in a datacenter for over 3 years now. X25-E, X25-M, 320, 520, DC S3500, DC S3700. Over 1,000 of them. Not one has failed. Just my experience though.

A few are getting low on their media wear indicator (25% remaining) but they are getting replaced due to capacity issues.

Most drives, when they "wear out", simply flip to read-only mode. At least you can get your data at that point.

Mechanical drives, on the other hand, are simply dead.

What do you do with the old ones?

I got a new Thinkpad with an SSD at my job last year and it died after 10 months. Luckily I had pushed everything but it was still mighty inconvenient. I just put one in my desktop and will be backing up religiously.

This is just one data point, I know, but I have never (knock on wood) had a hard drive of any kind fail before.

In our data center we have several thousand hard drives and nearly two thousand SSDs churning away day in and day out, indexing and caching Etc. So far the failure rates has been about 1/10th as much for the SSDs per-capita than the Hard drives.

Not definitive, just a data point. (oh and the SSDs are the Intel X25-M (160G version))

Data Recovery guy here...

SSDs are fast - but you better back them up. When an SSD fails its usually complete and without warning.

Spinning drives will often give you some warning signs, clicking, slowing down, bad sector errors etc.

Spinning disks can be rebuilt with new heads or platter swapped into a new body.

With SSDs you cant just remove the NAND chips and recover the data either. SSDs use dynamic wear leveling to shuffle pages of blocks around - in other words - block 0 is not at block 0 -it could be anywhere... and if the internal map is lost - good luck.

The data in the NAND chips is also encrypted... they do this to ensure that erased sectors are actually erased. Because of the wear-leveling routine spies could possibly recover data from unused sectors - the encryption makes this impossible.

And for extra fun - intel likes to epoxy the chips to the board - just to be sure they don't fall off....

I compare them to a hover-car, with wheels you can get a flat and still make it off the freeway... but with the hover-car your just a sitting duck...

On the Mac with Time Machine, I can lose at most an hour of work. Not saying that to praise the Mac; I'm saying that because that's how all computers ought to be.

This would be a swell time for Dropbox to offer Linus a free large business account so he can store everything.

I've never used it, but do you mean a Desktop with a permanently attached external drive to backup to? How does that work on a laptop?

Apple Time Capsule. Look it up

Ah! But then I'd have another Apple specific gizmo in my life :-). But it looks interesting nevertheless. Thanks.

Then you are probably the sort that wants to use rsync and the --link-dest option, which is about the same. (Except for the nice retrieval interface.)

You can make a new backup and it will only consume space and copy bandwidth for the files which have changed. It is cheap to keep N backups for large N.

I have a raspberry pi running http://netatalk.sourceforge.net/ However, it seems to get in a corrupted if I close my laptop lid mid-backup. I don't lose anything, I just need to run disk util on the backup image. Prior to that I was using time machine on USB and never had an issue. If you work plugged in you'll never lose more than 5-10min of work. It'll remind you every 7 days you haven't backed up.

I was looking into a solution for Linux the other week and http://www.rsnapshot.org/ works very similarly and is what fits my needs there.

The benefit of time machine is that excludes unnecessary OS folders/files by default and the OS installer has a simple interface for recovering the whole hdd from a time machine backup. I recently found time machien now has tmutil, a commandline tool for accessing time machine backups which really helps make managing them more transparent.

Yeah, Dropbox doesn't play so well with Git...

If you use Git on just one Dropbox instance at a time, I assume it works just fine. "One instance at a time" is no different than anything else you do with Dropbox.

I meant to suggest Dropbox Business as a continuous backup solution, not as sync.

problem is as soon as you checkout a branch, dropbox sees a load of different files and tries to sync them. Then you switch back to another branch and dropbox tries to sync again. Do it often enough and your machine grinds to a halt

Use dropbox as a remote (bare) repo, not a working directory

First thing I thought, Linus should evaluate a MacBook. If I remember correctly he once stated that the MacBook Air was the perfect machine for him, so it's not that far-fetched.

Running MacOS isn't very helpful when you are trying to develop Linux.

I ha(ve/d) a small external hard drive enclosure which holds/held two 2 TB drives in RAID-1 mode (recently tested that it’s indeed RAID 1: The enclosure left out smoke and one of the drives now works happily in a single-drive enclosure), attache(s/d) via eSATA and is used every hour by an rsync backup run using hard links. Every evening, I run another script that cleans up most of the older backups of the last week, keeping one per day.

So, yes, while at home, I also lose at most one hour of stuff, provided that nothing catastrophic happens that damages both the drive and the notebook – unfortunately until very recently, the internet connection to backup offsite was simply not there, and even with 50/10 MBit/s it will be annoying.

If you hadn't written your own script, I'd recommend you use rsnapshot [1] instead. It is based on rsync. You can set up a schedule with multiple generations.

[1] http://www.rsnapshot.org

Yeah, I stumbled over it less than a week after I had set up some nice script to delete the extraneous backups I didn’t need. At the time, my requirements for keeping backups were a little stricter, too, due to space constraints, which were not easily implementable using rsnapshot. I guess with my ‘new’ scheme of one backup/day to keep, it’d be much easier.

I find if vaguely disturbing that John Stoffel suggests to Linus that he mirror two discs in the future (presumably meaning a RAID-1 arrangement) and Linus responds with "I long ago gave up on doing backups."

Linus conflating RAID with backup?!? Surely not. My world feels turned upside down... >smile<

RAID is a lesser class of backup, and he's given up on the entire realm.

If you are doing something as important as this, why aren't you at least running some sort of RAID setup that would have been fine with losing a disk? Or at the very minimum had a decent backup setup so that you would not lose data.

For someone who created Linux in the first place, this must/should be very embarrassing.

OTOH, it must be nice to know that even if your house blows up (hopefully when you're not in it), your life's work is mirrored and backed up on millions of computers. All he lost were some pull requests which have to be re-sent. The submitters know who they are and if the changes don't show up in a day they just send another email - no harm, no foul.

This is especially ironic, as Linus has lately been extolling the virtues of SSDs:

http://www.wired.com/wiredenterprise/2012/10/linus-torvalds-...

And when Linus speaks, a lot of people listen.

Additional: http://www.phoronix.com/scan.php?page=news_item&px=MTQ1ODY

Maybe buy two hard disks, and back up one to the other.

It is only the linux kernel afterall.

*Linus' copy of the Linux kernel.

Yes Linus. Why don't you use RAID ?

In a laptop?

There are some laptops with two hard drives, but whether or not these models are nice to use, I don't know. Or maybe they have other hardware with poor linux support....

So... does this mean that Apple's flash storage is low-quality?

I use the same MacBook Air that Linus does (except it's a 13" model), never turn it off, have reinstalled the OS multiple times, archive my data, do around ~3GB worth of transfers every day, and run 3 different VMs more often than not.

Maybe I should just upgrade to a newer SSD. Anyone had luck with OWC?

All devices can fail. No use to overstate an anecdote.

Relevant Linus quote:

"Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)"

source: https://groups.google.com/forum/#!msg/linux.dev.kernel/2OEgU...

That still mostly worked (s/ftp/git/)

    > I had pushed out _most_ of my pulls today, so
    > realistically I didn't lose a lot of work.

You're always going to lose the things newer than your last backup.

That's why the hourly backups of a system like Time Machine is great.

I've had several drives fail, and I've only lost about half an hour of work, excluding that which was pushed, picking up almost exactly where I left off.

Reply on news.ycombinator.com