Also introduces that same time delay for getting security patches released by package maintainers into your build pipeline.
It may be safer to just accept that you live in the wild west and have systems boundaries in place to limit exposure/impact.
I've been pushing for closed-box logging targets, for example, where all logging goes to at least two systems (local and remote) so that the logging target is generally only available to submit log lines entries to. Not perfect, but where my head has been at.
Another thing is to PULL backups, not push them out.
That's an absolute must. A push backup may not be a backup at all when you need it, or it may be compromised. It also requires the system you push into to be accessible for inbound connections, which in itself may be problematic.
You want append only backups with secure timestamps and do forensics to find the last good snapshot. It is easier to set that up with pull backups, but not so hard with the right tools for push backups.
Where am I going wrong?
The main benefit of pull-based backups is that the production machine doesn't need credentials to write to the backup server; this means if production is compromised, it can't corrupt your backups.
Therefore, a push system is no different than a pull system, provided, of course, that the production system can only make new backups, not write indiscriminately to the backup server (e.g. delete old backups).
If production is compromised, you can't trust either.
> Therefore, a push system is no different than a pull system
Not entirely - a push system can DOS the backups much easier than a pull system (filling the disks, say), and a push system requires append-only backups in order to protect against backup corruption. A pull system just requires read-only access into production, which is much more simple to configure, audit, enforce, and maintain (IMO).
Yes, yes, yes!
Not only that, but test your backups! And follow the 3-2-1 rule for any data of any importance: 3 copies in at least 2 different formats and 1 copy located off-site and in a different physical location from the other 2. Be sure to check the integrity of your backups, as well. It doesn't do much good to restore from a corrupted backup.
That all sounds like a lot of work, and an excess of paranoia, but you won't be thinking that when it comes time to restore from those backups. Let's see: pay millions of dollars in ransom anonymously in BTC to hackers who've held your data (and your company!) hostage, or, spend anywhere from a couple of hours to a few days restoring from backups.
It's your call, but, I know which one I would choose if I were the one who gets paged for things like that.
Also we are cautious of using dependencies that don’t provide long term support/back fixes - where possible I pick stable, responsibly-managed dependencies. Postgres is a great example, security fixes are applied across multiple major versions, not just the latest.
Version pinning, self-managed forks and code reviews of the dependency on upgrade.
In particular, who is auditing the old version of the code that you happen to be running to make sure it doesn't have vulnerabilities that are now gone after a non-security-motivated change like a refactoring or a feature removal? Probably not the upstream maintainers, who generally only maintain HEAD.
It can be a pain but that pain might be motivation to not pull in dependencies with little thought.
Running private package infrastructure with audited dependencies isn't a panacea to stopping supply chain attacks. I do believe it's an effective defense-in-depth tactic for the reasons others have discussed.
An additional supporting tactic that should be done is to tightly control egress traffic. Like ingress traffic, all egress traffic should be denied by default. From there, traffic should be whitelisted. That makes it more difficult to exfiltrate data or communicate with command and control infrastructure. Tight control on egress traffic also makes it easier to alert on unexpected connection attempts. That all said, locking down egress traffic can be a pain. It also isn't a panacea. Where there’s a will there’s a way.