What if you encounter an nginx bug, or a kernel bug? At some point, when you reach a low enough level you will need some deeper investigation tools and finally access to the machine to get enough data and fix the problem.
But honestly, how often is a web applications bug due to a kernel bug?
I swear, sysadmins were annoying as an application developer, but devops is something else.
People with one year of actual work experience get hired as devops, and have all the privileges I would need to fix their mistakes, but I can’t, because I’m an ‘application developer’. So instead you end up teaching them how to do their job.
I’m not salty at all.
You can write the same screed full of generalizations from the perspective of any job title: a devops person would lament the fresh-out-of-bootcamp “application developers” who have no idea how systems work together so write SQL queries that retrieve a million rows, one at a time. “Works on my local!”
Saying the emperor has no clothes is not white tower thinking,
If your programmers aren't delighted with you, I'd say you the Ops person is not practicing DevOps or you have a buy-in problem to DevOps practices at an organization level.
My previous role had my title as "DevOps Engineer" but it always rubbed me the wrong way. I was just an Operations Engineer with a focus on making my developers' jobs easier, in any way I could. Having that as my North Star kept me honest about the work I was doing versus considering the role more like Operations Engineer v2.0.
In the Silicon Valley, at least, DevOps seemed to be (seems to be?) sort of in vogue; I think it's important to keep its core qualities of bridging Development and Operations in mind as opposed to just shifting an existing position's title in an attempt to attract talent.
And this should extend throughout the organization. If Architecture or Security or any other group is making your life miserable, they too should be DevOps'ing, working closely with you, caring about your frustrations that only they can fix. Sadly there are still so many silos left to break up.
Most people with such a title are actually something like “Automation Engineers”, “Infrastructure Engineers”, “Operations Engineers”, “Site Reliability Engineers”, etc, that are involved in a DevOps “process”, “initiative”, “culture”, etc.
However it’s also misguided to assume that specialities don’t exist. You can have infrastructure guys, developers, security folk; and there will be overlap between each role but it’s impossible to be the master of each trade.
I agree that arrogance is an unpleasant trait but arrogance can take many forms, rudeness to colleagues, or over assuming ones own technical capabilities in adjacent fields.
b) In my experience, the application developer is held responsible for the application's behavior in production. In the luckiest .01% of scenarios, there might be an infrastructure engineer with appropriate permissions and free time trolling the Slack support channel at the moment you report the issue. Otherwise 99.99% of the time infrastructure will not acknowledge of investigate anything complicated or subtle with just one service owner complaining. The infrastructure group, organizationally, is graded on shipping new platform features and on coarse KPIs for the performance of the platform as a whole; nobody is getting paid to investigate the weird bugs of some application team somewhere.
No, it doesn't eliminate such distinctions. My view of DevOps is more about ensuring that automation is used as much as possible to meet objectives.
It's definitely not about making everyone a homogeneous developer unit that can work on every problem.
People come in all shapes and sizes, some are more competent with certain things than others, others have a lot more experience with certain things. That's aside from the whole preference thing - not everyone wants to or has an interest in managing infrastructure.
Maybe when you have a handful of developers and a small set of infrastructure, that's fine - but at a certain point you start to require more and more specialised knowledge. Yes, even when you're all-in on Cloud and using all the SAAS/PAAS products out there.
>Otherwise 99.99% of the time infrastructure will not acknowledge of investigate anything complicated or subtle with just one service owner complaining.
Yeah, that's an organisational problem from the sound of it.
I think it’s best to consider the original source which is this talk from Flickr: https://youtu.be/LdOe18KhtT4
("10+ Deploys Per Day: Dev and Ops Cooperation at Flickr").
Directly it talks about joining developers and operations into the same team- later Patrick Debois would refer to this as DevOps and a year later the first DevOps days in Ghent was organised (also by Debois).
We really need some new terminology.
I'm being a little flippant here though, I am aware that developers are often asked to wear the sysadmin hat too.
You can't hire twenty developers that all have the same skill/inclinations, the same interests, the same experience.
That's not to say that a DevOps Engineer is some super 10x rockstar developer - no, they're going to have the same variations on skill, interests, experience, etc.
It depends on your environment, but there's so much different tech once you count the entire stack, that I don't think it's reasonable to expect any one person to be an expert on all of it, or even a lot of it.
> > Every developer needs access to some servers for example to check the application logs.
> I fundamentally disagree with this.
So,
Developers shouldn't be reaching for SSH access to check logs.
If you're encountering problems that you can't diagnose through the existing logs, then you should probably be involving at least one other person - someone who has that production experience, who might have some additional knowledge about the problem.
If, and only if you've exhausted other avenues - then reach out for SSH access. But it should be a last resort, not the first resort. Plus, anyone SSHing into production boxes should really be very familiar with how production is configured. You can do more harm than good by poking around on a production box being completely unaware that you're causing alarms and outages elsewhere because you taking a memory dump of nginx caused in-flight requests to get timeouts and so-forth. The people with that experience are generally the DevOps/Infrastructrue folks since they're the ones who deal with production all day, and are going to get the pages if something goes wrong with that.
Yeah, perhaps. It'll depend on the circumstances, right.
I've got a reasonable amount of GSuite and Exchange experience, and same for Active Directory. I'm reasonably confident that I can work my way around those and do most of what I need to do without breaking it.
I needed some GSuite groups set up, some folks added to them, and a GSuite OAuth application set up and some values passed back and forth to do some integration. The only way to do all of that is with full GSuite Administrator rights.
Now, I could ask why they don't just give all Devops folks GSuite admin rights, it'd be much easier (for me) and I could do my job more efficiently.
The response is going to be something along the lines of:
> You don't need that access most of the time. > The times you do need that access it's often for a limited time or scope, and to resolve a specific problem. > For now, It's better that you work with someone who is responsible for that stuff on a day to day basis to do those things.
This is, in my opinion, pretty reasonable. Sure, it meant more delay until someone was available to do the GSuite configuration, and we needed to jump on a call to pass IDs back and forth and test it out. But it got done, and it wasn't overly burdonsome.
They didn't hire me as a GSuite, Exchange or AD Admin - they already have folks to handle that. That I have that knowledge and experience is still useful for the company - I know exactly what to request is done, and we can talk on the same level about it. Heck, if it turns out that I need this every day and I'm constantly going back and forth with them on setting things up - then I might get that access, but it'll probably come with an explicit requirement that I use it in specific ways, that I follow their processes/procedures, and keep them informed on when I'm using it and why
There is also the case of ratios. An organization probably needs more developers in specific areas than DevOps, so with dedicated DevOps you could concentrate similar work from across several teams to a dedicated DevOps team that knows that work very well.
Giving everyone production SSH experience is, in my experience, a way to run into all sorts of weirdness, not to mention endless frustration.
In a modern automated infrastructure, that box is likely a container running on a virtual machine that's ephemeral and can (and probably will) go away at any moment based on any number reasons - maybe CD kicked off a new deployment, or maybe the load changed and the instance was selected for scale-down, or maybe our spot bid for that AZ isn't sufficient for keeping the instance around, maybe you being SSHed in and poking around impacted the health-check, and so it's being killed for not performing right.
Theres many other problems, too - lots of applications are built in some way that there's simply no other way than secrets (passwords, api tokens, keys) to reach other systems, particularly third party systems. So production boxes have production secrets, which you probably don't want to share with everyone.
Giving everyone SSH access so they can, in theory, take nginx/kernel dumps as needed tends to imply giving superuser rights, which means they can do whatever they like.
So, yes, pull in someone else - find some way to try and reproduce the problem NOT on production, if that fails, perhaps there's a way to grab enough detail or pull additional logs or network captures to identify the issue. If that fails, well okay, lets SSH in - but we need to coordinate that to ensure that instance does't go away, and doesn't impact production while you do it.
That is the responsibility of system administrators. Application developers have no business on a production machine. If your sysadmins don't have the technical skills to diagnose these problems, they are incompetent and must be replaced.
The actual result of this is that the sysadmins are not replaced, and the application developers end up in an emergency conference call at 3am to tell the sysadmins which buttons to click on the production environment, since they’re not allowed access themselves.
DevOps is not a role or role segregation, it’s about aligning incentives and outcomes across functions in an org (hopefully through collaboration, tooling, and knowledge transfer).
The caveat is that if your org is fundamentally broken, none of the above applies or works and it’s all lipstick on a pig.