Comment by XERQ | Hacker News Reader

Comment by XERQ | Hacker News Reader

I've seen many of my clients set up their own backup systems and have those fail at the worst times. Last month a large client of ours called our managed support team at 3AM saying they hired the wrong developer who completely trashed their database and hosed their entire application. They had their own backup system in place and it silently failed, but luckily they ordered our internal backup solution as a secondary. We were able to get them restored in 5 minutes, if they didn't have our solution in place they would've had to spend weeks fixing what the developer broke.

Current Linux backup solutions are not made for humans. Have a look at the mondorescue guide[1], nobody is going to read that and comprehend it with full mastery, meaning you're leaving yourself open to losing data. VPS providers offer backups that are usually in the same datacenter, which means you're SOL if there's a disaster. Those same providers also don't allow you to restore single files/directories from snapshots, usually you have to launch a new instance or revert everything back to snapshot.

[plug] We ended up creating a simple Linux backup solution[2] that's as simple as copying and pasting a single command to get installed, notifies you if your backups aren't running, handles snapshots, and is secure. Restoring your data is a single command away, so you can focus instead on building your startup rocketship. Our mission is to make data loss a thing of the past. [/plug]

[1] http://www.mondorescue.org/docs/mondorescue-howto.html

[2] https://jarvys.io

How do you handle a scenario like this...

I have a Redmine install that I want to backup. It uses both a database and the file system. If I attach a file to an issue, the attachment is stored on the FS with a reference in the DB.

I don't know how Redmine deals with keeping the FS vs the DB consistent, but assume it uses some type of transaction across both when an attachment is added.

How do you back that up without possibly getting the FS and the DB out of sync. Ex: What if you snapshot the FS right after an attachment is written, but before the reference is added to the DB? The transaction in the app will succeed, but your backup is inconsistent compared to what the app expects.

How can you get a truly consistent backup without either a) stopping the app or b) integrating with the app to make sure you're not breaking assumptions needed for consistency?

Basically, almost every backup solution I've ever seen is crash consistent at best. How is yours different?

I've been struggling with this myself in various cases. The conclusion I've come down to, is that the app has to take some responsibility to provide for consistent backups, either via checkpoints / write barriers, or via a quiesce command that can finalize in-flight transactions, and pause / buffer operations long enough for a volume snapshot to take place. The other thing I've seen apps do is provide a data export functionality, so you end up backing up the export file not the live data. But that requires extra time and disk space. Another option is to run the app in a VM, and do a live VM snapshot which includes the running state of the VM.

Bottom line is if you can't live backup your app, file a bug report with the app vendor. Although it would be really nice if there were standards in this area, so that a backup tool would just need to call one operation to put all supported apps into a hot-backup mode.

Our core value is around the simplicity of use, so instead of having to read a novel of a manual written by a crusty UNIX sysadmin, along with combining it with storage, you can create an account and copy-and-paste a single command that does all the work for you. We've even tested it with people who've never used Linux before and they were able to install it and restore a file without much direction.

With that said, we do provide the ability to hook your own scripts and commands into the backup process itself, for instance backing up MySQL you'd just put in the mysqldump command (with relevant DB and user/pass info) into the hook script uncleverly titlted 'run-before-backup.sh' in the config directory.

I personally don't have any experience with Redmine. Application-specific backups are outside the scope of our initial launch, but as we get more user feedback we'll be able to build plugins that integrate with specific applications. In the mean time, I'd both: 1) pester the developers to provide a simple backup solution for Redmine, and 2) look into either putting it on a VM that you can snapshot, or use something like ZFS snapshots and send/receive it to a remote location.

Um. From your docs:

C. RESTORE FROM ANOTHER SERVER'S BACKUP

This feature is currently in development.

So... My server goes up in a puff of smoke and I can't access the backup?

This is a beta product, and we put that up on the docs so people can understand in its current state what it's capable of and what it isn't. One thing we didn't put in the docs is that we can change the target of a new system to an old server's backup. So in the case of your server going up in smoke, you would simply open a uservoice ticket (or email me using my profile info), fire up a new server installed with JARVYS (disabling the cronjob until you're restored), and we'd change the target UUID to that of the new server, allowing you to restore from your old backups.

Soon we will have it where you can simply select any other server from your account to restore from it to any other server on your account, and the docs will be updated with how to do it.

This is why I prefer to have objects in the DB if they are not too large or numerous for that to be practical, that way the transactional integrity of the DB and its backup handles this. Unfortunately it is not always practical or under your control.

From a developers PoV you can increase the consistency of your users backups by being careful how you order operations. If you make sure the DB is updated after the file is in place and tell the app administrators to backup the database first, and your blobs are insert-and-soft-delete only (you can do updates in an insert & soft delete only way with versioning instead of in-place updates). That way you are never in a position where you have a missing blob, though you might end up with orphans where inserts happen during the time the backup took to run.

Other than that, as you say, you have to stop the app or the app has to have built in support for consistent backups (or integration into your chosen solution).

To minimise the downtime associated with this you can use tricks like LVM snapshots or your OS's equivalent. That way the downtime is only as long as it takes to stop the app, take the snapshot, and restart the app, instead of the full length of time it takes to make the backup (which could be a fairly long time if it involves a full backup of a large DB), and it means you can coordinate the backups of apps that are integrated (or just used in sync) but not in a transactionally safe way, without having to stop them all for the full backup run.

That is the best you can do without explicit support in the app - there is only so much a backup solution can be in control of.

> Last month a large client of ours called our managed support team at 3AM saying they hired the wrong developer who completely trashed their database and hosed their entire application.

Meaning that they didn't have a QA process in place before putting things in production, or that "dev" is also "production"?

That's surprisingly common for "internal" applications. And quite frankly, the type of dev that screws everything up like that is also the type of dev that doesn't know any better, doesn't version things, doesn't have an issue tracker and doesn't set up DB backups, etc., etc.

That's an institutional failure in this case. I certainly wouldn't expect a dev to setup backups for the company database, for instance. A VCS and issue tracker should be a company-wide (or at least department-wide) thing, not something you setup (or not) from project to project.

Reply on news.ycombinator.com