Hacker News Reader

Comment by andmarios | Hacker News Reader

Indeed, but ops people have to learn new technologies all the time, so it's not that big of a leap. When you talk about a cloud and see a sysadmin rolling their eyes, it's not that they have something specifically against the cloud, it's that they've seen so many diverse bugs and issues in their life, that they know you just exchange one set of problems for another.

Also, a challenge to using clouds more effectively is you need to bring onboard your developers. Ops people often have to act as coaches and basic systems' knowledge is very important in this role.

View on HN

It’s not just about learning a new technology it’s about learning a new mindset. Unfortunately part of that mindset is giving up control over the underlying infrastructure, automation, and could be making yourself or some of your redundant.

Not too many ops people that I’ve dealt with that move to AWS reach for Python, CloudFormation, creating custom resources with lambda to plug into CloudFormation, etc to automate. Most won’t even use System Manager to automate EC2 maintenance.

I've seen a good number that do once the org makes it a priority and allowed money to be spent on learning and experimenting.

From there, some people still want to stick to their old knowledge, but chances are you still will need that. Only start ups are "easily" going 100% Serverless. And even then many still want a few 24/7 servers. So there's still a job for older ops.

Ops people tend to be slower because brand spanking new tends to be over hyped and the brand new thing was designed with a goal for ease of use and performance. Security will be added on perhaps next year. Systems manager is cool, but if you already need to use something else to orchestrate, it's often easier to use that and keep a smaller toolset.

It’s not about going serverless, it’s about how to automate the management of servers. When my manager first came, the one ops guy we had was manually going through a list of EC2 instances and patching them. That’s an old mindset.

We have a third party web application that was on one server and when it went down, they would get an alert and reboot it. That was an easy win - put it behind a load balancer and an autoscale group with a minimum and maximum of 2 and http health checks so it would just kill the instance and bring a new one up.

This same guy saw a lot of EC2 instances that he didn’t recognize come up in the middle of the night and wanted them to follow our naming conventions APP0x and wanted to know the IP addresses. It took me forever to get him to understand that they were ephemeral, were part of an autoscaling group based on a queue, and that he couldn’t put whatever old school alerting system on them and they were all tagged with the corresponding autoscaling group and environment. The best I could tell him was the subnet that they would be launched in and the corresponding CIDR block.

Just curious - if the ephemeral instances are caused not by regular usage, but a mining botnet, will you have the tools to detect what is wrong (before the bills skyrocket) and mitigate?

That’s interesting. I guess we could set up something that used a combination of CloudTrail/CloudWatch Event/lambda that monitored any EC2 instances that are launched without the required tags and alert someone.

We do have billing alerts set up already that would warn us something went wrong we would have to investigate to find out what.

Better alerting and monitoring is one of our goals this year.

Heck I’m still finding places with hard coded keys in programs instead of letting the SDKs get the keys from the default configuration file (development) or from the instance role when they are run on EC2/lambda. Unfortunately all of the examples do it and the developers didn’t know any better.

Reply on news.ycombinator.com