Hacker News Reader

Comment by tracker1 | Hacker News Reader

This... worked in an org, where it literally took nearly 3 months to get package updates and additions approved... and that was for licensing review, not even proper security audits.

It may be safer to just accept that you live in the wild west and have systems boundaries in place to limit exposure/impact.

I've been pushing for closed-box logging targets, for example, where all logging goes to at least two systems (local and remote) so that the logging target is generally only available to submit log lines entries to. Not perfect, but where my head has been at.

Another thing is to PULL backups, not push them out.

View on HN

> Another thing is to PULL backups, not push them out.

That's an absolute must. A push backup may not be a backup at all when you need it, or it may be compromised. It also requires the system you push into to be accessible for inbound connections, which in itself may be problematic.

Really once the secure channel between two processes running on each host is setup, there is no difference between the two methods. A pull backup can be tricked just as easily by presenting a false filesystem or backdooring the 'pull process' running on the compromised machine.

You want append only backups with secure timestamps and do forensics to find the last good snapshot. It is easier to set that up with pull backups, but not so hard with the right tools for push backups.

Another option is to use a cloud storage service that supports fine grained access control (e.g. write-only vs read-only) and retention polices.

Re: backups

Yes, yes, yes!

Not only that, but test your backups! And follow the 3-2-1 rule for any data of any importance: 3 copies in at least 2 different formats and 1 copy located off-site and in a different physical location from the other 2. Be sure to check the integrity of your backups, as well. It doesn't do much good to restore from a corrupted backup.

That all sounds like a lot of work, and an excess of paranoia, but you won't be thinking that when it comes time to restore from those backups. Let's see: pay millions of dollars in ransom anonymously in BTC to hackers who've held your data (and your company!) hostage, or, spend anywhere from a couple of hours to a few days restoring from backups.

It's your call, but, I know which one I would choose if I were the one who gets paged for things like that.

I'm trying to wrap my head around the concept of 'pulling' backups, rather than pushing them. In my mind, once you make a backup, you should then transfer it to a separate system for archival.

Where am I going wrong?

To pull backups, the backup system connects to the production system and grabs the data, storing it locally on the backup system. To push backups would be for the production system to connect to the backup system and send the data.

The main benefit of pull-based backups is that the production machine doesn't need credentials to write to the backup server; this means if production is compromised, it can't corrupt your backups.

If you can’t trust the production machine to initiate regular backups by itself, why do you trust the production machine to allow the backup server to access the production machine and make backups? In both cases you need an alert system to detect if a production system has not been backed up for too long.

Therefore, a push system is no different than a pull system, provided, of course, that the production system can only make new backups, not write indiscriminately to the backup server (e.g. delete old backups).

> If you can’t trust the production machine to initiate regular backups by itself, why do you trust the production machine to allow access by the backup server?

If production is compromised, you can't trust either.

> Therefore, a push system is no different than a pull system

Not entirely - a push system can DOS the backups much easier than a pull system (filling the disks, say), and a push system requires append-only backups in order to protect against backup corruption. A pull system just requires read-only access into production, which is much more simple to configure, audit, enforce, and maintain (IMO).

Reply on news.ycombinator.com