Ask HN: I'm being left out of SRE and DevOps by my manager. Should I be worried? | Hacker News Reader

Ask HN: I'm being left out of SRE and DevOps by my manager. Should I be worried?

15 pointsDrBGB27 comments

I am a web developer working on react and backend REST APIs. I have the skills and want to be closer to the deployed product working as SRE and DevOps. Writing anisble scripts and maintaining production instances.

It's a small team. I'm not allowed to do Devops things because 'me being a developer is worth more'. I think that's a terrible idea. I can only become a better developer by working close to the deployed product.

Having less SREs and shared devops knowledge can only hurt things. The team rationale is computers fail all the time. We should have SRE scripts that recover things. I hate that idea. I want to know why servers failed. My belief is if you don't understand the root cause it'll just keep happening.

Is this a naieve belief? Am I in the wrong here?

View on HN

> 'me being a developer is worth more'

See things from your manager's perspective. They have work they need accomplished. They have a small team - maybe not big enough for the work they need to accomplish - and of that work, some will directly create profit for the company and some will cut/ control costs, and still other work will be "stuff".

Your manager is telling you that they need your skills in a particular area, like due to staff shortages, skill shortages, or more specifically the value of the work. Take this as a sign that your skills set and labor are valued, just not perhaps in the way you prefer.

If developing skills in other areas (IE: SRE, devops, etc) is important to you, consider ongoing conversations where you make it clear you want to build this skills, and partner with your manager to do so. If that just isn't going to happen in this team, then consider your career path elsewhere if this is really important to you.

A lot of developers have a cowboy coder mentality that causes problems when they are given access to production instances. That's caused many organizations to establish a wall between development teams and operations.

Learn to work with your devops team to get the information you need. Developers should not need direct production access.

> Developers should not need direct production access.

Agreed with the proviso that there are enough people this is possible (not always).

I'm a senior dev who runs a team and I am the only one with access to production which I've not used yet - if I ever actually need to use that access then the outcome will be a discussion on how to avoid that need in future (my access is for a holy shit, that should never have happend, put it back up and figure out how it happened now scenario).

It's not just a stability issue, it is also a data protection/security issue - I trust every dev on my team but if they don't have access whether I trust them or not is immaterial.

> Developers should not need direct production access.

Then stop telling me I need years of experience with Docker, K8s, AWS, etc. on dev job listings.

The architect who designed my house does not need access to come inside my house anytime they feel like it.

This is a poor analogy. Does a developer need to understand some infra fundamentals in order to properly architect applications? Absolutely. Should they need as much experience as a devops of infra engineer? Absolutely not. That's just shoveling more work and expectations on someone to squeeze more value for the same dollars out of them.

There are only so many productive hours in the day. You can not be an expert in everything, there simply isn't enough time.

> The architect who designed my house does not need access to the construction site.

FTFY

Architects routinely visit job sites. Of we continue to twist this analogy, staging would be the construction site and the finished building would be prod. Developers never need access to prod.

It’s also worth noting that if you’ve got enterprise customers, and you’re shipping “on prem” to them, your company might not even be able to access “prod.” You need to get the devops to a point of maturity that you can hand the whole stack over to an engineer at the customer company and have them take it from there.

If in order to get a job as a senior architect I have to have experience that I can only get by having access to come inside your house, then yeah I kind of do.

Nope. Developers can get all the experience they need in a dev sandbox environment with all the same tools and infra. They do not need access to the production environment.

It's a responsibility / authority issue, generally brought on by poor management. I get what you are saying, and I suspect many in this thread agree with your sentiments. Frankly, when I'm wearing the developer hat I'd rather not have production access, because then I can tell management / the business that I can't fix their production issue. But, often management is going to cajole, yell, scream, etc at whomever they think can fix the problem, and that often ends up with a developer who doesn't even want to be there having production access in order to fight yet another fire. Shouldn't be that way, but there it is.

I remember seeing job postings in the early 2000s requiring 10 years of php experience.

Those requirements are usually BS added by hr or a recruiter mangling what devs said they need on the team. i.e. "it'd be nice if they were familar with aws" becomes "aws required... well we're looking for someone above jr level, so all the requirements need at least 10 years..."

Devs don’t necessarily need to be involved with IAM roles, but they should know what their app requires.

Devs that have their work deployed on k8s don’t need to know how the ingress controller is configured, or how etcd works, but will be crippled if you’re confused my service and deployments.

All that being said, job listings are often filled with nonsense. If you like the company, just ignore all of that and start a conversation.

Just because I don't have direct production access doesn't mean that the people with direct production access aren't just executing the same scripts with different environment variables that I'm writing and running in the development and test environments.

I write code for many projects. I vastly prefer those that forbid my access to the production instance. So much less stress.

> The team rationale is computers fail all the time. We should have SRE scripts that recover things.

This is a potentially very risky approach. I was given the task of having my org stand up a large, complicated system. My engineering team built a system that was failing 2% here, 10% there, 5% somewhere, etc. They built recovery and retry mechanisms for every failure scenario. We ended up with a combined failure rate of 40% that was masking the failures but basically doubled our execution time for the service. I locked down all new development and directed all of our effort toward driving accumulated failures below 5%. They completely disagreed with me (because they'd gotten used to the life of being firefighters). Once we got to below 5% their opinion changed completely. They're very proud of how they were able to drive down the error rate.

Watch out for accumulated error (or even worse--compounded error)!

On the other hand, you can expect that any network based service will fail and need to retry. It's especially important to make software that relies on cloud-based services resilient in this regard.

"Not being allowed to do DevOps things" is best practice, and sometimes mandated by law or policy ("zero developer access to production.")

Some possibilities for doing DevOps things again are:

* request a "glass break" account for emergencies. You'll need to meet with someone senior to get that signed off on, and then someone needs to write some automation for that.

* Ask to move to a DevOps role instead of back end web dev.

Some possibilities for better visibility, which is a better goal if you ask me:

* set up a status meeting that's once every week or two weeks or month with the DevOps/SRE team or person(s)

* ask to be part of the monitoring setup, or set up your own monitoring that you can then share.

If we transcend the holy war of dev vs ops, practically speaking this is likely a team culture and skill set question.

If you being a dev is worth more, the team either has the ops under control, or you need to demonstrate why you being closer to the ops provides business value.

If your org treats dev as more valuable than SRE you likely aren’t doing SRE correctly. It should be a fairly high status role. Your org likely just has an old fashioned ops team that has rebranded.

In any case you definitely don’t want to move from a high status position to a low status one in your organization. Switch orgs to one that treats what you want to do as high status.

For what it’s worth, I agree that your manager is wrong but there are lots of ways to skin a cat and a dev/op split is a very traditional route that lots of teams are successful at.

> We should have SRE scripts that recover things. I hate that idea. I want to know why servers failed. My belief is if you don't understand the root cause it'll just keep happening.

It sounds like you are not only worried about you being involved, but also about the current state of your team's DevOps / SRE practices. Hopefully you can still push for higher standards in that space even if you are not directly involved day to day.

> It's a small team. I'm not allowed to do Devops things because 'me being a developer is worth more'. I think that's a terrible idea. I can only become a better developer by working close to the deployed product.

By the same token, it is not good for you to be the expert on everything. With a small team it's easy for one person to become the go-to SME for everything. This is just as dangerous as having faulty DevOps and SRE practices. The company has little ability to prevent you from leaving, getting sick, forgetting things that only you touched, etc.

I think it's pretty common, as a developer, to be told, effectively, "Get back in the corner and do what you were hired to do -- not think -- just code -- leave the grown-up things to the grown-ups."

This is just an extension of Taylorism -- early assembly-line thinking -- where each human becomes like a very simple machine -- and therefore stupid, dull-minded, unquestioning.

That said, I think you are conflating several different issues here.

But just sticking to your first question in the title -- no, you do not need to be worried.

Unless you decide you want to be more than a coder/developer. In that case, start looking for a new job.

> me being a developer is worth more

so a salary increase is justified, then, eh?

I'm having the opposite problem. My organization (Fortune 500 company) has decided to fire almost all of the sys admins and say that developers are responsible for their full stack. My company usually has teams of 1 developer each, and each developer has several projects. Given this arrangement I have to not only write all of the code (mostly scientific computing and statistics) and mange the servers (OS updates, deployments, etc).

What were the responsibilities listed in the job you applied for?

>Writing anisble scripts

Skunkworks an effort on this in your downtime and in doing so prove your capabilities, would be my suggestion.

>maintaining production instances.

If you're deploying to AWS or another cloud provider it's more than rational that it's locked down to a certain set of people (devops/SRE) made responsible for that infrastructure.

>I want to know why servers failed.

Ask for read access to your logging infrastructure?

You're not wrong. You're just working with the wrong people. If you want to be a part of the full system including SRE and DevOps tasks start looking for a new place to work. If you expect this group to change you're probably setting yourself up for failure.

Thank you for this reminder.

Reply on news.ycombinator.com