Hacker News Reader

Comment by brobinson | Hacker News Reader

>2048 bit RSA keys

Github recommends 4096 now, for what it's worth. [1]

>Pushing database backups offsite

This is a really bad idea and a good way to get owned. Database backups must be PULLED from the server, not pushed from it. Separately, you also need to test that you can restore from your backups periodically.

There were a couple other things I disagree with, but they're in the realm of personal preference. It's also interesting that this guide uses Linode which is a hosting company known to have had quite egregious security issues in the past. [2]

[1] https://help.github.com/articles/generating-a-new-ssh-key-an...

[2] https://www.google.com/#q=linode+hack

View on HN

I agree that pulling does appear to be more secure, but if someone breaches any of your servers undetected, you're pretty much hosed regardless.

Plus, the Inversoft guide specifically states that backups must be encrypted. I could put my backup ZIPs on a public Github repository and no one would be able to access the user data stored inside it. Therefore, it really doesn't matted if they are pushed or pulled.

If a script or cronjob or whatever on the server has write access to the repo, so does the hypothetical attacker.

I remember seeing a "hacking" website get wiped along with all of its backups about 15 years ago and it left a very strong impression on me regarding this issue.

If it is write access but not delete access (nor overwrite access) then there is no issue.

It is sometimes more secure to push because it requires no inbound connections or authorizations the live machine.

That may be true but with a pulled db backup and a good recovery strategy you'll be back up relatively quickly, without you'll be DOA.

>This is a really bad idea and a good way to get owned. Database backups must be PULLED

Just make sure your database server doesn't have permission to delete backups (e.g. Have it POST a backup via HTTPS). There is nothing wrong with db server initiated backups.

This is still open to abuse in the form of your legitimate backup script being disabled and replaced by one uploading false backups. You wouldn't know anything has happened until you go to restore from one.

Of course, this is an edge case, but I believe setting up a pull-based backup system is still going to be less work than a write-only push system.

A hacked server can have the backup server pull false backups as well.

Of course, your servers shouldn't be SSHing to your backup servers, but that goes both ways.

If the backup machine is compromised the attacker can corrupt your backups. Either way, if machine is hacked you risk loss.

That's ridiculous. The same tricks can be played on a pull based system once you have compromised the DB server.

Why must they be PULLED?

If a server can push backups, it must necessarily have credentials for and write access to that backup repository. If the server is compromised, that write access could be used to attack the backup repository as well. Pulling backups from a server removes the need for credentials or access to the backup repository.

That is one valid way of solving this problem. Another option is to have a permission system that allows you specify that the server can only append new data, but not delete or overwrite existing files. This is useful if you don't want the added complexity of having a separate "backup management" service, which could easily become a bottleneck if you are backing up a lot of data from multiple servers.

One way to do this is with S3, for example, is to use an IAM role with only the "PutObject" permission, and enable object versioning for the bucket to prevent a compromised server from being able to delete data by overwriting existing files.

It's also easy to do this with Tarsnap. You can split out your host keys with tarsnap-keymgmt into separate read, write, and delete keys. Just take your delete key off of the host and store it somewhere secure.

Or better yet, create the key somewhere else and only upload the keys you want to have on the server. Otherwise there's a chance that the original tarsnap key is left in an unallocated block.

Not an expert here, but this seems to me to be stripping away one layer of security. Kind of like saying that you can leave a key to your house lying around as long as the pantry door is locked - if your database server gets owned, they now have some kind of login to your backup server as opposed to no kind of login.

Maybe the extra layer doesn't add much security, but if it's a simple config change and it does add something, wouldn't it be worth doing?

Not a bad point - it's a trade-off.

You could make a similar point about the centralized backup management server - it needs to have "some kind of login" to all your production database server, so of that host is compromised (which might only store encrypted copies of your backup), so are all your database servers if those privileges can be escalated. You could argue that one backup host is easier to secure than a complex system such as AWS, but then I would argue that it would probably be hard to beat the track record of S3/IAM. ;-)

Both approaches have their place. If you're dealing with a large number of hosts and a lot of data, the backup host will quickly become a bottleneck and you're probably better off with the append-only approach. If that's not a concern for you (i.e. you're not running into any bandwidth limits on the backup hosts) and want to deal with operating yet another service to avoid the risk of e.g. IAM privilege escalation, the other solution might be a better fit.

Doesn't entirely fix the problem, but using public/private key auth and a private key created just for the backup cron job, the server can push a backup via scp/ssh to its own user account which on the destination end is chrooted into a particular backup destination directory.

Plus, if you're super paranoid, pulling doesn't require leaving any ports open and software listening on the repository.

But it requires leaving a port open on the DB server where an attacker can get a whole copy of your database, or destroy it.

Many database servers are not serving databases solely to themselves and have that port open as part of the normal course of operation.

Could you recommend another hosting service with better security practices than Linode?

I don't currently use them, but I've had good luck with Vultr and they haven't had any incidents like Linode AFAIK. There's also AWS, Azure, whatever Google's thing is called, and all the other big ones. I have no opinion on DigitalOcean even though I currently have a VPS with them since I'm not running anything remotely mission critical there.

Reply on news.ycombinator.com