Hacker News Reader

Comment by mypastself | Hacker News Reader

How does everyone here handle the patches and updates on their network? Do you perform updates when you remember to, or do you have regularly scheduled sessions, or did you (heaven forbid) automate it?

The more home servers and services I have, the harder it is to keep everything up to date…

View on HN

Ubiquiti recently had a horrible update to their AP firmware that made the devices unusable with cryptic symptoms. Users would attempt to connect to the Wifi SSID and just get rejected.

This burned me so bad that I have to second guess updating software now, which creates a moral hazard. We need to figure out how to price and actually extract the externality cost that these errors create. Otherwise, we are in a perpetual gradual slide.

tldr double the critical path hardware budget.

> We need to figure out how to price

What is MSRP on extra set of hardware that is physically disconnected, as failback.

> and actually extract the externality cost

Roll it into overhead.

I put my trust in the lord above and go bravely into the night.

We don’t look at porn on our network so I believe he will be pleased and grant us protection.

I use Ansible to update machines en masse. List your machines in your inventory file and then run `ansible $GROUPNAME -m command -bK -a "$UPGRADECOMMAND"`. Groups are especially useful if you have machines with different Linux distros that use different commands. I understand it's even possible to use Ansible to manage Windows machines.

Of course, now you have to be really careful to guard access to your Ansible setup...

Use '-m package -a state=latest' and it should upgrade most distributions without needing to care about the underlying managers... I think. I could see some oddities with package name resolutions!

Access - don't share your private keys!

We have defined weekly and/or monthly maintenance windows scheduled for this kind of stuff. Nobody on the user side ever remembers these, but at least in theory this is a time when systems can be expected to be down for patching and updates, or other maintenance/upgrades.

I set up a systemd timer that runs `pacman -Syu` every morning, and I keep an eye on the Arch Linux mailing list. It's worked out pretty well over the past few years. There's only been one or two times where something broke to the point that I had to roll back an entire machine.

Core network infra is typically redundant such that supervisor modules can be upgraded “hot”. Leaf networking attachment to the hosts should be redundant such that taking out a single path to apply updates to half the fabric should similarly be a non-event.

For my home network I do it once a year. My nano pi r4s and tplink access points all run openwrt but the amount of updates they put out is so quick and the update process not seamless that it ends up becoming a pain.

I wonder if ubiquity does this better.

docker and watchtower solves this problem for me. Automated updates sound bad, but usually updates rarely go wrong. Even if some update does, the risk of being compromised far outweights the happy feeling of 99.999% uptime green bars.

For my family's personal machines that I have control over, a periodic todo.

Reply on news.ycombinator.com