Hacker News Reader

Hacker News

Comment by joshstrange | Hacker News Reader

Full thread

joshstrange

Honest question, what would you call a role that:

* Is on call

* Manages internal software (grafana, Prometheus, salt stack, etc)

* is the first line of defense for issues in the field, works with support and the engineering team to handle problems

* Manages a distributed fleet of servers (uses off the self and/or custom code to do so)

* Builds internal tools/automations to improve the reliability of our platform and/or address problems automatically

* Pushes us to make software/OS updates

I ask because that’s what I’d love to hire. Someone who owns the production environment and has the time/space to focus on improving/maintaining it.

“Ops” or “Production engineer” is what I’ve referred to it as in the past but this seems like a good place to ask what others would call it.

View on HN

As long as that very description is spelled out clearly near the top of the job ad, it doesn't matter terribly much (within reason) what you call it - job searchers will try various different strings to find it. Personally, I'd call it an ops role, however.

In general, most of the issues I've seen with these sorts of roles aren't the naming of them but rather the third bullet in your list. Having an ops role that's responsible for all the problems with software they didn't build and weren't allowed to have meaningful input to the design/implementation of isn't healthy. It sucks up their time and energy fighting fires they didn't create and likely aren't really empowered to fix. This is the problem with this role (as often implemented) that Rachel talks about in her first paragraph.

If you really care about having a good and reliable product you need people involved with the design and implementation that are deeply invested in making it reliable and maintainable in the long term - which means either making your dev roles shoulder some of the oncall burden or having people that straddle the ops and dev teams. Or both. If you're having difficulty filling this role, perhaps this is the real problem?

I haven’t started trying to hire for this position yet, it’s been something I’ve been thinking of for a while though. Right a small number of developers, myself included, are on call and I’d like to reduce that a bit or just share the burden.

My goal isn’t to throw crap over the fence and say “make it work” but rather empower someone to make maintaining and growing our platform their main goal. The developers (again, myself included) are not great at the ops side of things and can rarely focus on the infrastructure itself due to other priorities (yes, we can talk about how that itself is an issue). If I could clone myself and one of specialize in ops and the other on programming for the platform I would in a heartbeat.

Infra/Ops and programming are two different mindsets (much like managing people or qa differs from writing code). Switching between them is hard and you pay a penalty to do so. Not to mention there are skills (networking is high on that list) that I’m not good at. I can scrape by but that’s not where my skills lie. That’s why I’d like to hire someone who is good at it, who _does_ enjoy it, and who push for changes from a ops perspective that I can’t due to time or skill.

> Infra/Ops and programming are two different mindsets [...]

HARD disagree on this. I've done both. Most of the really excellent programmers I've worked with have, at least a little. You can't write highly reliable networking software without a deep understanding of how networking actually works. You can't write highly performing software without a deep understanding of the infrastructure and hardware it's running on. And so on.

I'm not trying to bersmirch yours or your teams abilities here - if you're writing in a high level language and most of your challenges are implementing biz logic then not knowing very much about the underlying infrastructure and hardware is fine, you aren't trying to write a distributed RDBMS, you probably don't need to know this stuff.

But do remember that there are lots of people for whom the hardware, the infrastructure and the application they're writing are inextricably linked. It's not a different mindset, it's just people with additional skills you haven't needed to learn yet.

Note that I said mindset not skillset.

I have zero doubt that I could do an Ops job well, what I can’t do is switch between writing business logic and maintaining servers at the drop of a hat. Similar to how I can’t go from QA to engineer without a context switch/penalty.

As I said elsewhere in this thread if I could clone myself and do both roles I would in an instant. But if I have to pick one I’d pick writing code (as my primary thing), not saying that Ops doesn’t write code, just not the same type of code.

For me it's very similar, I was in dev, now I'm in ops and I can easily switch back. But to think that all good programmers can do infra is a falsehood.

I've met so many developers who don't even know how computers really work. They're good at a particular tech stack and do their job very well, but they can't do much else. Let alone infra.

Personally, I agree that it's two different mindsets, but sometimes they can overlap.

I've settled on "systems engineer" -- I just try to keep "systems" in the title to imply that my job is largely -- "making many things work together to run the business"

I think my current title has the word "cloud" in it or something and my boss rejects (with prejudice) "SRE" or "devops" explicitly to avoid being labeled as operations.

At our organization, a bunch of devs which were interested in improving/automating infrastructure split into a separate DevOps team. They first called themselves DevOps, then renamed themselves to SRE, then again to Production. Every time they changed their name they prepared presentations for the whole company explaining the name change and why the previous name wasn't correct after all. Everyone is still confused and always forgets what's their current official name.

Other than that, there's also:

- a separate IT department which manages networks, telephony and access

- the ones who push people to update software/OS is InfoSec, a separate team

- ordinary developers can make changes to k8s/grafana/Prometheus configs for their services, if it's reviewed by Production as the owners (they're understaffed compared to the multiple dev teams)

- there's also the "Core" team which deals with the microservice platform on top of bare infrastructure (how microservices should communicate, etc.)

Often the lines are blurred and it's not always clear who you call if you have a problem.

"Platform Engineer" has started to grow as a name for pretty much this. An engineer that builds the platform that all things run on. For example, the k8s clusters, the observability stack, has on-call, and builds the internal tools and automations (called the IDP, or internal developer platform).

IDP already has a very important meaning as a part of any platform :)

Yea, it's confusing, i give you that.

Apparently not everyone gets it. I once went to an AWS meetup and introduced myself as a "platform engineer", but not everyone understood what I did =)

But personally, I like "platform engineer" more than "ops".

What I've experienced working well is to consider platform/infra as just another dev team. They should experience the same good practices (staging changes, tests, documentation, clean code) and duties (level 2 on calls, regular postmortems from ops, etc).

Ops/prod/support eng are generally more business related and less technical than infra/platform. They make sure the processes run as they should, or operate with agreed procedures when something is raised. Some issues may be lightly technical ("theres an alert on free disk space of server X", "provider Y changed their SFTP keys without notice", etc) and some business related ("provider Z is late to push updates", "it's a holiday on country of provider A", "service of team B raises an exception when situation C occurs"). They are often level 1, in between the actual dev teams (escalating issues to them, or asking for better stability, logs, resiliency, etc) and the platform/infra team.

Being faced with production issues is a burden for focus, etc. I wouldn't want my platform team to be on level 1 handling these stuff or nothing get done. Still, I wouldn't want them to be completely free from it or they would loose track with reality, just as any other dev team.

To summarize, I would say:

- A fleet of business dev team - A fleet of platform / infra team - A support/ops/prod team handling level 1 through procedures written by dev/platform teams, and raising to infra or business dev for level 2. Regular feedback sessions with each other team to have them stay on touch with real life. Some light coding / improvements tasks paired with infra or dev to get them a better understanding of the underlying layers.

From the description you provided, that's what I would call a "platform engineer". Though the titles are sensitive for some people, and there's a lot of discrepancies between companies, so i usually let people from platform choose the title they prefer, between SRE / DevOps / platform eng / cloud eng / infra eng.

>what would you call a role that …

Overworked?

It used to be systems administrator, or network engineer, depending how deep into infrastructure it goes.

That's reliability engineering, IMO.

Reply on news.ycombinator.com