That stuff gets archived (zero compression) with recovery data; with PAR2 recovery data as well; it then gets copied to multiple USB sticks and microSD cards. These get posted out to various family members (who are asked nicely to keep them somewhere safe) and kept in my firesafe.
Any passwords and serial numbers and etc are printed out, with the name and URL of the software they're for. A copy of that goes in the fire safe. There are copies of the spreadsheet on various drives.
All the other stuff is kruft. I do have multiple copies of my music but that's only because tagging MP3 files is hell.
I'd be interested to hear how the drobo failed.
Put some cash in the fire safe. If someone breaks into it (thinking there is money) they will probably be less likely to take other things in the safe or destroy those items (I mean a hard drive they might leave - obviously they will take anything of clear value). Assumption also is that the HD is encrypted.
This comes via a practice a long time ago with cash registers. You leave the cash register unlocked and put some cash in it. Thief is presumably happy and doesn't trash the rest of the place.
"Any passwords and serial numbers and etc are printed out, with the name and URL of the software they're for. A copy of that goes in the fire safe. "
Doesn't seem like a good idea to put plain text passwords (is that what you are saying) in a place that someone will look thinking there is something else of value in there.
The typical fire safe is meant to prevent auto-ignition of paper. That means that your media could easily be subjected to temps up to 400 F -- maybe higher!
You've controlled for this by sending them offsite as well, but most ppl won't!
First off, if you're in the market for a data recovery on your Drobo, I strongly recommend getting in touch with Gillware Data Recovery: https://gillware.com/. Their prices are very fair, they have great staff, and they have a system that lets you preview your recovered data before paying anything. We got a new business client entirely because his RAID 1 NAS pooped itself and Gillware was able to recover everything he needed from it.
If you've got a lot of data, online backup services are usually not an option. My usual advice then is to have a completely separate device that automatically pulls data from your primary device and stores backups in an archival format. If you have the time and inclination to do the setup on it, I really recommend BackupPC for this: http://backuppc.sourceforge.net/. I've used BackupPC for years for larger-scale backups; it's very reliable, it's not too difficult to get going, it does deduplication and compression and pooling and uses rsync (when possible) to reduce network load, it can easily be configured to send you an email in the event of trouble, and it has a simple web interface you can check into on a regular basis. I've yet to find a commercial shrink-wrapped product that does what BackupPC does as well as BackupPC does it.
The downside of course is that you still wouldn't have offsite backups, so in the event of a fire or other local disaster, you might still be looking at complete data loss. If at all possible, try to pick a few of the most important directories of files, and get those backed up online. I haven't heard so many bad things about Crashplan or Backblaze, so I'm curious what you've heard.
I use this to backup:
rsync -vaxXES --delete --backup --backup-dir="/Volumes/backup/__rsync/$(date +"%Y-%m-%d_%H%M%S")/" --log-file="/Volumes/backup/__rsync/$(date +"%Y-%m-%d_%H%M%S").log" --exclude="__rsync/" --exclude=".Trash/" --exclude=".Trashes/" --exclude=".Spotlight-V100" --exclude=".fseventsd/" --exclude=".DocumentRevisions-V100/" --exclude=".TemporaryItems/" /Volumes/data/ /Volumes/backup/
Sometimes i delete stuff in the __rsync dir manually.
I've tried Crashplan and Backblaze, but they're crazy slow from Sweden.
rsnapshot (http://www.rsnapshot.org/) from other machines (they are Linux as well) onto the Linux backup server (so now there are two physically separate copies, master machine and backup machine). And rsnapshot provides "point in time" views of the backups.
Periodically rsync the rsnapshot tree to an offsite location with enough space to hold the 220G worth of backup data.
What I don't do, which you imply you did with your Drobo, is have the only copy of anything truly important on the RAID5 array only. If Linux md or LVM corrupts the backup partition I just remake it, and start rsync'ing again (and pull old snapshots from the offsite). If one drive fails in the RAID5 array, I'll replace the drive and rebuild the array, but the data should still be there. If two drives fail in the RAID5 array close on each others heels (so that the whole RAID array goes away) then this is the same as if md or LVM corrupted the data.
"Backup" does not mean having "redundant protection" in one device, it means having plural separate, fully independent, copies such that corruption/loss of one copy is meaningless to the other copies.
People replying will recommend many backup solutions. No matter what you go with, it's important to trust no single party. Your home could burn down. Dropbox/Crashplan/S3/etc could fail. Media in a safe-deposit box could become corrupted over time. (This is common with flash-based storage and CDs.) Pick at least two independent backup solutions.
Also, backups are a lifestyle change. They're something you have to maintain indefinitely. File systems, I/O interfaces, media formats, APIs, and businesses all come and go over time. There's no one thing that is sure to keep your data around for decades.
Finally, most people don't have much data that's actually important. I have maybe 20GB of stuff that matters to me: photos I've taken, documents I've written, and a few videos. In terms of value per byte, documents are the highest, followed by photos, then video. It's much easier to maintain backups when your data set is small. Music, books, and other mass media can be re-bought/downloaded.
Using paper will keep your data around for decades.
And analog media has many disadvantages. Storing, reproducing, transporting, searching, or modifying analog stuff is really hard.
- Avoid Backblaze unless you have your external drives connected at all times, files are erased after 30 days if not seen by the app.
- GDrive is the cheapest, but can't backup external HDDs at all.
- Arq/Glacier or Tarsnap is probably the safest archival you can get, but can be quite expensive depending on your usage pattern.
- Crashplan, as Backblaze, focus on 'devices'. They are best suited to people who don't have lots of files outside of their computer. Crashplan has a terrible reputation for losing customer's data.
In the end I went with Bitcasa[1] and am quite happy so far. It works the way I want it to: not necessary to mirror all data locally, can throw files adhoc at it, is accessible anywhere, decent web UI, mobile apps. Price is fair (€8/month for 1TB) and speeds are decent. It's the best I could find, still keeping my HDD backup in a closet just in case...
[1] http://l.bitcasa.com/XFLTXjG7 (with my referral code)
I have 1 (1T) hard drive at work, to have an off-site backup.
Plan to use Arq for important files to back on Glacier.
I'd change the two hard drive system to a 4 bay RAID setup using Synology etc.
----
I was thinking of buying Drobo but I guess I was always scared of their "secret sauce" and getting stuck with recovery if anything happened to it.
I've tried online backup services and was really disappointed by the throughput. I think it took me nearly 5 months to do an initial backup on Crashplan and that didn't even include my photo archive, VMWare images, etc.
I've tried using services like Flickr as photo backup but the retrieval process is brutal. There's no good way to bulk retrieve photos from them.
I do use Dropbox to sort of back up anything I'm currently working on but they don't retain history past 30 days without paying for Packrat. Plus I've soured on them due to privacy concerns.
I do use BitBucket for almost all of my source code and I've been pretty happy with it.
I do use Time Machine on an old 1TB drive for my main system (MacBook Pro) and that's actually been pretty damn good. The handful of times I've needed to recover a file lately, I've been able to get it from Time Machine in a couple minutes with little hassle. I do need to get a newer, bigger drive for that sometime.
Wow! I forgot to mention but I do this too! Thank you! My wife has called me paranoid of the highest order and now I can point to this comment and feel better that I am not the only one.
>> I've tried online backup services and was really disappointed by the throughput
Yes. I have been disappointed with all of these services.
>> Flickr as photo backup but the retrieval process is brutal.
Yes. I tried apps like Flickery etc but it is just not worth it.
>> I do use Dropbox
Nice idea. I have some files on it but I only use Dropbox (the free version) for files that I need on-the-go for anywhere-access.
>> I do use BitBucket for almost all of my source code and I've been pretty happy with it.
Same here. Bitbucket for some private random experiments and projects, Github for things that I don't care about.
>> I do use Time Machine on an old 1TB
Actually, Time Machine is the only decent backup utility I have ever found. Ever. The ease of use is just mind-blowing. I remember more than a decade ago, when my only chance was to wipe out the system and reinstall the OS or to keep a partition or separate drive for home/data. And with newer OSX, Time Machine can back up to as many drives as you want, which makes it easy for me to take actual direct backups on multiple drives.
On top of it, privacy is such a big concern. Then, it usually becomes a pain. I tried https://www.aerofs.com/ too but then left it because it was too buggy. I will have to try it again.
- Configs and small important files: http://www.tarsnap.com/
Laptop:
- 2 x TimeMachine
- Amazon Glacier with Arq (Weekly, just important files)
- External Disks mirrored to same model with CarbonCopyCloner
The other two disks get crammed into USB enclosures, and I use rsync to copy data around in a manner very similar to jwz's http://www.jwz.org/doc/backups.html. With USB 3.0, this is very fast.
That keeps me from experiencing a catastrophic data loss scenario, which is step one. It also solves a work continuity problem wherein I would otherwise be left now with the problem restoring from backups. Here, restoration is literally a matter of swapping disks and rebooting.
Do I actually lose data? Well, yeah, but I've put an upper bound on it that is livable: I lose at most a week because the disk in my drawer at work gets swapped out every Friday for the one I keep at home.
Some caveats: I don't use encryption at the level of my OS's VFS layer (yet), and I don't use RAID. I also don't use non-standard configurations like HFS+ with case sensitivity, btrfs, ZFS, etc.
I keep most sensitive data (think private keys) elsewhere. Those are backed up by way of dumping to hardcopy and storing with my tax information. I can OCR the ASCII-armored representations later if I need to, and it makes choosing a fire safe easy since I don't need one that's rated for digital data storage.
Now, this unfortunately doesn't solve the problem of storing historical copies of things, and I'm trying to find a way to accomplish that sanely without having to resort to alien technology like Plan 9's venti and fossil or vendor lock-in à la Time Machine, or rsyncing to dated directories.
├── Code
├── Dirty
├── Downloads
├── Media
│ ├── Books
│ ├── Music
│ ├── Pictures
│ └── Videos
├── Projects
└── Scripts
I try to keep it very tidy because otherwise I would create a mess.
"Dirty" and "Downloads" are working like /tmp, except they're getting wiped every 5 days [1].
Everything except "Downloads" and "Dirty" is backed up daily, incrementally and encrypted with s3cmd [2]. Some Inspiration here [3]. Sometimes I delete old stuff manually with s3cmd or from the S3 backend.
Dotfiles and bootstrap scripts are stored on Github and Dropbox, so are serious projects.
I also keep track of installed packages to be able to restore state on a new machine [4]. I'm not very emotional about restoring my machine. With a fast internet connection it takes me like 20-30 minutes to completely restore a wiped machine.[1] http://askubuntu.com/a/20831 [2] http://s3tools.org/ [3] http://www.triatechnology.com/encrypted-incremental-backups-... [4] http://askubuntu.com/a/137991
My rules whenever anyone asks about backup:
1. Use backup systems, not file storage. Copying files somewhere is not a backup system. You could delete or corrupt files and never notice. Backup systems keep historic copies of files.
2. Use at least three backup locations. For example: Local hard drive, machine in a different building, the cloud. CrashPlan lets you do this, for example.
3. Test your backups regularly. It's boring, tedious and tricky, but it's less painful than losing your photos.
The best photos are usually backed up on Facebook/etc. anyway.
* personal projects are both on github and on tarsnap
* my pictures are on google drive and on tarsnap
* my music goes into google music (even thought now i'm using spotify now)
everything else is either on google drive or in any other kind of cloud service.
You get the first 100GB for free + 50GB when you invite friends.
I'd love to answer any questions or concerns you might have about our service.
I also duplicate things to Google Drive. This happens automatically when I am on any network.
Doesn't protect against a fire. Or someone who swipes the router and/or the data source.
Suggest either an offsite backup or a fireproof hard drive (iosafe for example) to backup the backup. (TC is funky with attached drives or at least that is what I have found when trying to use an attached drive on an airport extreme). (Of course someone can also swipe the fireproof hard drive).
Also, I don't care (much) about old versions of my personal data. If I did, this would be an unacceptable backup solution because fire/theft of my TC could wipe it out, as you mention. But between my TimeCapsule and my Google Drive, it strikes me as unlikely that my laptop, my TC, and my Google Drive will fail me all at the same moment. If so, I would be SOL.
I set n at 2, and substitute a cheap "fire safe" for the safety deposit box ( I know, I know... ) . I don't know that I will build another RAID when this one goes. I have yet to pull anything off it.
But my primary copy of things is all still on my main machine, un-RAIDed. For legacy software reasons I won't bore you with, that's still a Win7 box.
Time Machine for documents and projects - those usually live in Dropbox or Github/Bitbucket as well.
I don't bother with backing up OS/configuration, since it takes no more than a day to get a clean OS and applications to a state where I'm feeling at home.
You could also look at setting up a NAS.
http://en.wikipedia.org/wiki/Network-attached_storage
I don't' trust any backup source that is not my own.
I have 1263776 files or 582GB backed-up for only $60,00
All servers, laptops and desktops I run occasionally put their files onto my main-desktop disc and everything gets uploaded to carbonite.
I have all my data in a separate partition, reaching 1.5 TB. With say 2 MBPS upload speed, it seems it would take about three months for the first upload.
Also, did you face any issues with data limits from your ISP for the first upload?
No trouble from my ISP. Being from the Netherlands, I obviously have a 100/100 fiber connection. So it really isn't that much.
Also, in my experience their throttling sucks. If I get back from traveling with a bunch of data like pictures or video, it crushes my upstream bandwidth and disrupts VoIP.
That said, for the price, it is an excellent service, and their restore mechanism works well.
Using TimeMachine on my laptop as well (for convenience).
I've got nothing but praise for Backblaze. Been using them for about 4 years and it's worked flawlessly.