Hacker News Reader

Comment by vosper | Hacker News Reader

Perhaps someone could give me some advice?

I work alongside a small team maintaining quite a lot of machines on AWS. They're struggling (IMHO) to manually apply all of the security patches their scanning tool identifies. My theory is that Amazon Linux gets patched frequently, and so they'd be better off spending time normalizing our EC2 infra so that every instance is running Amazon Linux, and then work on an easy rollout mechanism to deploy the latest version.

Has anyone got any thoughts on this? It wouldn't obviate the need for patching completely, but I feel like AWS is already doing some of this work for us, so we should take advantage.

View on HN

For those few AMI's that are long lived, AWS SSM Patch Manager is your friend. Naturally take care to roll out patches in a rolling block, you don't want to apply a broken patch everywhere in the same day :)

https://docs.aws.amazon.com/systems-manager/latest/userguide...

I second this, we use it to manage a bigger fleet with a few hundred machines. One thing to keep in mind though is that it will not apply kernel updates (as those require a reboot) so you still need to account for it.

I haven’t tried this yet (our instances need other changes to get to this point) but AL2 can support live kernel patching: https://docs.aws.amazon.com/systems-manager/latest/userguide...

Every mainstream upstream Linux vendor is continuously pushing updated AMIs. It shouldn’t really matter whether you solve this with Ubuntu or Amazon Linux or RHEL/CentOS.

Sounds like you need a better process / automation for rolling updates. Either continuously rebuilt golden images or rolling security patches, or turning on your distros unattended upgrade mechanism could be solutions depending on your environment.

Recently we moved to use EC2 Image Builder, and it's been working great: https://aws.amazon.com/image-builder/

Note: We were already mostly Launch Templates/ASGs, so updates are always new instances (rather than patching long-running ones).

For the life of me I never got Image Builder working in a decent state.

I opted for Packer and I've been very happy with it. Though with that said I'm still using AWS SSM Patch Manager for a few outliers that are long lived.

Like. You, Okta AD agent that can only be programmatically installed using AHK. :-/

It was a little strange to set up, I remember it taking a while/a lot of experimentation... But in the end it's just running userdata, and/or "component" scripts, and baking that into the AMI. It's been happily updating and switching out Launch Template versions for our ASGs (for reasons each pipeline can only push to 5 LTs).

I guess I should write up a blogpost, because... the documentation is kinda garbage.

I never got around to using packer properly so can't compare.

Yes, one of the core benefits of a provider like AWS is that they provide tooling to treat individual instances as immutable entities that you simply replace without any interruption to your users. You should focus on expressing the infrastructure as code and using mechanisms like ASGs to roll out new instances based on the latest Amazon provided AMIs.

If you can, definitely standardize on as few distros as possible. It'll make applying patches (and learning when things go wrong, because they will) much easier.

We used to have all sorts of distros that people just felt like using without worrying about their maintainability. We kept fighting fires to keep everything running. Once we standardized on a single distro (CentOS at the time), everything started working much more smoothly. We could have picked Debian, Ubuntu, it doesn't matter.

That being said, Amazon Linux 2 is pretty well maintained. Most things (all?) that work on RHEL, will work on it. You may need to use 3rd-party repos if you want really newer stuff (eg. PHP) but that's inherent to such LTS releases. That situation is expected to improve with the improvements that adopting Fedora brings in AL2022 but I need to catch up.

Yep we do this, works good - you can either trigger a server refresh from SNS (AWS notifies you of certain AMI updates) or we just rebuild our underlying fleet each week with the most current AL2 AMI

Reply on news.ycombinator.com