Difference between DevOps, SecOps and DevSecOps
five9sclub.substack.com
five9sclub.substack.com
It doesn’t tell me anything about what you do; and it implies that sysadmins never used to code either.
There are people pushing “devops tools” but those tools used to be called deployment or build tools. The word is pointless ambiguity.
Additionally: the “title” was coined as the name of a conference to include developers. The actual intended job title (and original conference name) was “agile systems administrator” according to the person who invented the word (Patrick Dubois).
Absolutely, i call myself adaptable Administrator ;)
In others companies it means separate siloed DevOps teams and people, You need to raise request to do anything on your servers. Sometimes deployment mean getting DevOps team approval in process.
I also dislike DevOps terms because it is not specific enough on how you should organize Dev and Ops concerns.
I'm always upset when I see a installations that are created with pattern: "we need it to run and we doesn't care where system will be in next two years" (in trashcan, usually - or it will act as a jumphost for another scam/DDoS). And I see such systems everyday. And that dramas, when I ask a simple question like: "how it will be upgraded"? "What is your plan about dealing with manually-installed python modules that overwrites files from system-installed ones when system upgrade will be performed?" and so on, and so on...
Fortunately a separate "devops" team that really cares about infrastructure reliability usually evolves into a normal sysadmins in no-time and their priorities are usually different that rest of "dev-teams".
changes increase instability and unpredicatability in environments. and the people keeping them running 24/7 are usually the ones getting paged when shit hits the fan.
Long-term maintanability is especially hard, and the lower you get in the OSI model, the more painstakingly it becomes to replace things without major impact.
It does, but to be honest, when I studied computer science "not coding" was a big deal.
There were many people who simply didn't like to do it for whatever reason, and those weren't "theory people". There were meetups on how to structure your studies so you would get your degree with at least code to write as possible, and most of those people focused on getting jobs in operations later.
For some of them DevOps was cool, because it finally allowed them to learn to code in an area they were interested in, but for others it meant their low-code jobs transformed to developer jobs over time, which they didn't like at all.
I was intimidated when I learned but that was a little bit because my first language book was COBOL and I didn’t even have a working compiler.
The “good” sysadmins I learned from were all able to code, in fact many of them used to be developers themselves.
And not just because of the language features, but also because of the community around it.
JavaScript isn't pretty, but it has a big community.
The Rust community today might be more helpful than the C/C++ community back in the days, so learning Rust might still be easier than an easier language.
I started with C and Java which I both didn't like much, but learning to code was a big wish for me as long as I know computers exist, so I worked through the problems.
A good friend of mine suffers dyslexia and dyscalcula, and was just never able to jump past the standard unix shell scripting skillset, not for lack of trying. His understanding of Unix internals, TCP/IP, DNS and TQM-oriented process development is mind-blowing.
He just retired from a successful, 30 year career architecting and implementing secure messaging systems for several large banks. If you bank with Wells Fargo, Bank of America or Wachovia then you've probably communicated with your bank through systems he built.
Back when Luke wrote Puppet, he himself admitted that he was a mediocre developer with a good idea. His premise was that most SysAdmins weren't developers and that they would be more effective with a DSL that wrapped a templating engine and an executor then really learning how to code to build their own tools.
This set just an ugly trend. Next came Chef, which was written by someone (Adam Jacobs) who was pissed off at Luke because he felt entitled to 100% of his time. This was an incredibly dramatic exchange, Puppet Issue #1010 (I can't seem to find any of the old tickets anymore). Adam was making bank off of Puppet consulting, Luke was barely scraping by.
So Chef came along, it was supposed to bridge the gap between Puppet and developers. It was also garbage, but after a few years you could sort of kind of do real development on it.
Then we got Ansible, meh. Salt Stack which was more for managing api driven saas infrastructure but had/has a lot of the bad architectural decisions made by Puppet.
Now we have Terraform, which is OK, but also garbage. On top of that there's the Terraform CDK, which sort of lets you code, but is really just a golang-like DSL that generates HCL or Json. We also have Pulumi, which like the TFCDK, is a golang-like DSL that .. generates HCL to be run by Puppet.
All of this garbage because Luke felt that Sysadmins couldn't code.
If I had a euro any time I heard a sysadmin tell me 'they can't code', only for me to see them write perfectly good Python... A euro for every time a sysadmin tell me they really want to program more, but they think they're underqualified to even start learning, even though they write higher quality, more maintainable and better designed code than most fresh CS grads I've met... But they're stuck writing YAML and HCL and maybe some shell scripts, because that's what the industry tells them they are qualified to do.
Years and years of gatekeeping 'real programmer' jobs behind silly algorithm trivia and CS degrees is to blame, IMO. Ansible-style DevOps is more of the same, pretending that programming is too difficult. So here, have this cursed undebuggable, untestable DSL that is totally not a bespoke programming language.
One thing that really helped my teammates and friends was asking them to do code reviews of my work, and them realizing "hey, if I understand this, I can fix it, if I can fix it, I can write it!"
Another was Ruby. Sure, Ruby sucks, but it's easy, almost anything you did just worked. This was why Puppet was written in Ruby (Luke spent months trying to do it in Python, then when he tried Ruby he had a working prototype in 2 or 3 days). Ruby was a great transition language.
And if i had an euro for every sysadmin who told me "i'm not a developer, you do it", i'd be rich. "Sysadmin" includes Windows-only, GUI-only, ClickOps style admins, which still exist. Do you think they can or want to code? Heck, there are even networking admins that prefer to use GUIs over CLIs!
Not everyone can or wants to write code, even if some people underestimate their skills, some people are afraid.
Judging by how popular Powershell has become in these 'GUI-only' ecosystems: yes, a lot of them can and want to.
But I'm not saying everyone always should code. I'm saying that not only wrangling Ansible and Terraform is already programming, it's a difficult kind of programming (no tests, no debugger, usually directly on prod), and that we should stop pretending that it's not. That the industry should stop it with the self-fulfilling myth that sysadmins don't want to program by continuing to build 'DevOps' tools centered around lame DSLs. Give sysadmins real programming languages, libraries and guidance on how to program, not shrink-wrapped tools that pretend they're purely configuration driven yet implement a full-blown programming langauge on top of YAML.
DevSecOps seems to just be plugging sonarqube into ci pipelines and installing siem/wafs.
Is it really better anywhere else?
Apart from the name mismatch and that it could have been just "Ops".
You can compare it with how we/they are doing agile.
You have to convince developers that they're responsible for making software easy to deploy and debug, and get them to carry pagers during work hours. You have to convince them to make the product internals clear for everyone, including to operations people, and get them to do knowledge exchange with them. You have to enable them by providing insight into production deployments and providing a platform that unifies production/development as much as possible without tons of boilerplate required per project (common database interfaces, common logging interfaces, common CI, common release procedures, common development environment).
You have to convince operations teams to write programs in something other than bash, and to generally follow good software engineering practices, to write operations-level code that is as testable as the application itself. You have to enable them by giving them the means to build a company-wide production platform where complexity is decimated by moving every product team onto the simplest possible setup. You have to let them build bespoke tooling and accept that these lines of code are as important as the application code itself.
With all of this implemented well, the differences between ops and dev roles will begin to blur, people with different backgrounds will mingle within and across teams, and you will end up with a more platform/product split between teams rather than a development/operations split.
But seriously, you hit upon something which is important, but very hard to communicate/admit - a big part of making the "DevOps Transformation" happen - which is a cheesy term but it conveys something better than just DevOps - is saying _no_ to devs and traditional SysAdmins and giving them a better alternative. Unfortunately this means that SysAdmin skillsets need to be supplemented or even supplanted by SWE skillsets. But the TL;DR is yes, doing DevOps often means putting the brakes on fun.
I am not a "DevOps" engineer. I am a digital platform engineer, with specialization in writing code to connect things to other things.
When I talk to people outside of tech, I just tell them I'm a software engineer. It's close enough.
Thank you for all the buzzwords without which poor sysadmin old me would've never heard of coding, automation, security, version control and so on. /s
ProdBizSupportUxGitDevSecOps - one person doing everything (or maybe I am msising some other Ops)
On a more serious note I think there is a trend to ask a lot of deep knowledge from one single person in too many areas.
I also find the definition odd: "GitOps is a way of implementing Continuous Deployment for cloud native applications" [0]
At least DevOps is more generic and not JavaOps or DotNetOps or PythonOps. Also I would have nothing against naming things like this except that along with naming things like this a bunch of hiring people starts asking for this very specific skills as it is like basic literacy that everyone should know or (a little bit worse) suddently appearing 1000 courses and materials from 0 to XOps in 29 minutes and suddently Github and Twitter is flooded with small projects copying at infinitum the same exercise without being able to actually teach people the hard skills of solving problems and then translating solutions into a specific technology(ies).
I hoped it was a parody. Making up fancy names for ordinary things - I wonder if it's hubris or immaturity in the industry.
Like, when you've done your homework properly and your production deployment system is automated, ACLd, auditable, programmable, well documented and reliable then adding something like an IRC/Slack bot to it is trivial to the point of being mundane. It ends up just being another trivial integration, another tool/frontend, and more of a quality of life improvement or something cutesy than something revolutionary.
When you haven't done your homework and your system is a mess, then yeah, integrating with IRC/Slack is a journey, requires you to use some superstar ChatOps tool from GitHub, requires you to spend weeks integrating your five different cloud providers into it, and you end up spending so much time on this that you have to turn it into a philosophy to justify the effort. But maybe, just maybe, you should focus on other things first. And maybe you shouldn't end up centralizing all of your deployment workflow on a third-party chat app.
That's not a bad thing and there's meaningful security testing that can be done with that strategy, but it's hard to cover all aspects of security testing in a fully automated DevSecOps pipeline.
Ops in all three of these becomes the modifier. This is why most practitioners today will tell you that DevOps/DevSecOps/SecOps are less individual titles and more a cultural way of thinking and operating and that the philosophy is executed by engineers of all types.
It's important to remember the origins as well. Prior to DevOps, developers and operations teams were siloed in ways that hindered healthy growth in todays distributed systems and cloud environments. These "*Ops" terms were born of the necessity to impress upon people the importance of interoperability between traditionally isolated and independent departments.
*Ops is the natural progression of Conway's Law into modern engineering organizations.
SecOps = sysadmin has to apply the CIS/SCAP/whatever security template to the machine
DevSecOps = developer also has to apply the CIS/SCAP/whatever security template to their machine
It all ends up becoming marketing jargon for suits looking to follow the next big trend.
Sysadmin is a System Administrator, which performs operations manually.
Creating a tools, that allows us doing a simple, repeatable and - usually - automated tasks, always was an important part of sysadm role. Of course, there were ones that did everything manually and wasn't able to write own code: we called them "operators".
From my point of view: "devop is that hasty one, that doesn't care about long-term support of underlying infrastructure".
Yep.
> Creating a tools, that allows us doing a simple, repeatable and - usually - automated tasks, always was an important part of sysadm role. Of course, there were ones that did everything manually and wasn't able to write own code: we called them "operators".
Yep.
> From my point of view: "devop is that hasty one, that doesn't care about long-term support of underlying infrastructure".
Routing is completely automated for instance, and has been for over 35 years.
For decades before DevOps was coined, Sysadmins were automating infrastructures with tools like Kickstart, Jumpstart, CFEngine, ISconf, Rancid, netboot/pxeboot/dhcp, etc. I first provisioned infrastructure with CFEngine in 1996 or 1997.
So, you did DevOps work in 1996, long before the DevOps term was coined.