What I Do as a DevOps Consultant
ilhicas.com
ilhicas.com
- Forget what the books said, theory, semantics and idealism. You are a DevOps Engineer, working in a platform specific team.
- You deal with Jenkins or similar, and consult other teams or developers to write scripts for it, or worse, do the work for them since they're too busy doing actual programming.
- Contrary to what they say, other than tiny scripts you are not programming much anymore, unless you consider configuration management to be programming.
- You are not allowed to come up with your own abstractions. Proposals to develop anything are automatically declined by your Product Owner, who simply points you to whatever tool with a fancy logo he could find listed in the Cloud Native Computing Foundation.
- You pride yourself for being an engineer, but decisions end up being only made in terms of getting free open source labor and ease up hiring, so you end up doing what everyone else is doing.
- Half of your team co-workers are perfectly fine dealing with churn, software updates and other sort of manual, classic sysadmin tasks.
- You are repeatedly being paged at night for problems that your fellow developers won't feel responsible for. You hear buzzwords and talks about this SRE thing or "you built it, you run it", but it never seems to materialise due to politics.
- You are getting a feel that cloud providers are not really making things easier nor cheaper.
It used to be a lot of fun back in the day, and a great chance to get paid to solve interesting problems related to system and infrastructure engineering instead of the boring CRUD work that plagued the last decades, but seeing the "App Store" that this turned into, if you are a creative individual with a software engineering background and you're considering a career in DevOps, my advice is to run away while you can.
I’m an ops engineer but develop the code to do my team’s operations tasks.
Well there's your first mistake. DevOps is not a career, unless you are a management consultant.
At current job, its kind of frustrating in that ee act as if we're doing dev ops, but as a dev, when a (my) job fails, ops doesn't trust me to either abort or restart the job.
So we've got all the buzzwords, none of the benefit.
Typical support call: ops calls me with the failed job. I log in, take a look see; tell them to download a file and restart a job. Whole lot takes an hour because I dont have permissions in prod to do anything but look at the log. Take that definition of devops and whove it up your ass.
I do recognize its not like this everywhere, but thays how it is at my current company, and they want to call it dev ops. More like roadblock he'll.
- deal with management who wont pay for a support license because the product is "Open Source"
- spend weeks troubleshooting dependancy hells for applications with no documentation and no architecture and listening devs tell you that "it works on their laptop" (but somehow they can't give you a working requirements.txt or equivalent)
- the immense pleasure of using apt, pip, npm, and all of their cousins behind corporate proxies
- deal with information security departments who have no idea what you are doing, and thus feel threatened and block all of your initiatives by default
- deal with infrastructure departments who are building their own internal cloud infrastructure, with no budget (hello again free "Open Source"), and won't label it "prod-ready" after 4 or 5 years while support for their legacy is dropped
- convince management that they may have to give some training to their 60 "integrators" and 80 "exploit" teams that are still copying and pasting shell scripts from Word documents written last decade or earlier
I’ll be starting a devops role next month, hopefully it will not be like this.
A sort of extension of the Lean System where workers from different areas share a common goal of delivering value to the final customer (in production), the same card may require a backend dev, frontend dev, and an ops to share the goal to take this card from idea to production, and then monitoring how it behaves in production.
For people who have not read the same books, "DevOps" mean things like "applying dev method in the operation field" ie. with ansible and the likes, coding the infra. After all, that's what that word seems to mean at first glance.
This reminds me of the term full-stack, some people consider it's just about being good at both frontend and backend, in my opinion a good full-stack is also a good networking engineer as well as good at customer acquisition.
For best ROI:
- integration must be automated and continuous
- delivery must be automated and continuous
- tech debt includes untested code and being late in dependency versions
- the above should be treated as impediments
The purpose of "DevOps" being to deliver faster from idea to production, it does necessarily include a toolchain of CI/CD/IaaS & monitoring tools. From this "DevOps toolchain", taylorists opened positions for developers that can maintain it around a software, they called that position "DevOps", which explains how we got there on having different meanings for the same word.
I highly recommend "Continuous Delivery & DevOps: QuickStart" by Paul Swartout for a first quick read on the matter.
This is pretty much how I sum up what you'd have to be to be "a devops engineer": an actually-full-stack developer with a strong facility for communication and for process.
Such folks are hard to find. I only know a few, out of the dozens of folks I've worked with.
While I have some deep knowledge in a few areas, most of the work I do has a lot more to do with researching tools, tracing down problems, and figuring out solutions that do not always appear on Google or StackOverflow.
That mindset has also served me well when I write backend code.
Science: because a scientific protocol is to be used when making and testing code, to understand every detail, and because it's based on logic
Art: because code should first be written for another human (or you in 6 months) to read and understand, that it works or not is secondary,
Sports: because you need everyday to be in the retrospective of your past performance and try to improve it
Completely false. A full-stack developer applies JavaScript frameworks to create websites. A DevOps engineer applies development methods to infrastructure i.e. infrastructure-as-code. Totally different both skillset and mindset.
The idea that a "full stack developer" is "hurr, JavaScript" but that devops engineers are somehow different is...questionable. I've shipped mobile apps, web apps, backend APIs, and build and operate computing infrastructure at scale in cloud and on-premises environments, while having also had success at the strategic level with regards to the development of and the pursuit of business requirements and objectives.
Write code to tell computers to do stuff. That's all any of it is. The elaboration of the details is important in the small but not important in the large.
I think it's more of a criticism on the term "full-stack" for someone who works on a backend and touches front-end every now and then.
I think everyone is in agreement that what we call "dev0ps" devs and "full-stack" devs have completely different skill sets and mindsets but if I had to attribute "full-stack" to either of these it would be a "dev-ops" dev. (theoretically, real life is messy)
I could say a real-full-stack should actually include EE, chip design, compiler design, kernel hacking, model checking, machine learning, virtual reality, power supply design, and 20 years of experience in Rust.
Or anything, really.
A solution can be technically fantastic, and yet people may not adopt it for one reason or another. You have to get buy-in from people.
I recently interviewed at a place where they vaguely know they needed devops but there wasn’t sufficient buy-in. I made the mistake of taking the interviews at face value. I should have been treating it as consultation from the get-go, drawing out any issues they might have (by starting with pain points), coming up with solutions. I didn’t realize I can be applying the same skills I have while working to at the very earliest stages.
If I were to build a consultation firm, I would sell organizations on the human interaction (because that is easier for people-managers to grasp; it is what they fo for their jobs). However, if I were hiring someone within my hypothetical consultation firm to do this kind of work, I would look at their problem solving mindset and people skills, at whatever experience level they are at.
*And it seems reading that Small Farmer Journal article about how to get started as a farmer with no money, real farmers have to problem solve too.
Why do you say this? This only deflates the meaning of the word impediment. To me it would only create ambiguity when we start using the word impediment for things that are not impediments to the common goal. How would you differ between those?
A: If yes, it is an impediment.
You say nothing to negate the assertion that the parents statements impede the organisation.
* Systems knowledge through the lens of Conway's Law: what does an ideal even look like in terms of organizational efficiency
* Leadership to get disparate teams in an organization to cooperate instead of compete
* Technical competency to help disparate teams adopt the principles, because telling people on the fence to "go Google it" is a plan for failure
People who get bogged down in CI/CD/IaC/containerization miss the forest for the trees and it's why most organizations never see any value out of their DevOps initiatives. If you don't know what to automate then WIP increases your costs. If you don't have leadership on board then you can't reorganize teams to prevent "cycles" and re-work inside the "factory". If you don't have technical competency then you have people re-inventing the wheel, poorly.
You need all three.
Did I mention we are information security consultants and work in neither dev nor ops? Yeah he wants our powerpoint slides checked into change control. I'm not joking.
Elsewhere in the thread, 'somepig notes that this sounds like what a sysadmin does. I replied there because, just as you shouldn't call yourself a programmer[0], calling yourself a sysadmin is probably similarly mindset- and career-limiting, but I kind of have a bigger beef with calling this "consulting" than I do "devops". Of course this stuff is "devops", but--"properly" applied (and I scare-quote the word because you can call damned near anything "devops", up to and including "the development team hurls a tarball over the wall at some sysadmins in the basement and flees at top speed")--it's not consulting. It's a list of hands-on work. There's only a cursory nod to the most important part of any devops-facilitating role (whether it's called an SRE, an "infrastructure engineer", a "devops engineer", whatever): alignment of technical initiatives with business needs.
I don't mean to be harsh in saying this, but if you're spending most of your time in the code mines grinding at this or that, you may not be a great "devops consultant". You might be a great infrastructure developer or SRE. But in my neck of the woods at least, clients hired me to make sure they were doing the right thing and to make sure that their employees were adequately prepared for the workflows they wanted to implement, not to do last-mile CI/CD computer touchery. (Which I'd do as well, of course, when time allows--but there's more value in being a multiplier than an adder, and at the rates I charged I'd better be the biggest multiplier I could.)
[0] - https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-pr...
This blog post really reflected my background, with most of the tech used being the same. However, most of the experience is by cleaning stuff up. I see stuff being used too many times that are not performant, effective or incorrectly configured due to people not taking care of said tech. Especially in the Javascript business. It is cleaning up tech debt that might not make cash in the short term, but many companies have it as a burden in one way of another and someone willingly to clean that mess up is very valueable, imho.
Cleaning up the results is sometimes due to a soberly considered "we have to move fast now to not die" decision, but often is just because it looked good in a five-minute demo.
I’ve always been comfortable with the idea of DevOps as a role. If you work on Terraform, CI/CD, container platforms etc and have knowledge of both Dev and Ops practices and engineering mindset then it seems like a fine label.
We really took a bet in the early days that DevOps as a discipline as well as an operating model and culture would emerge. We are now approaching 400 people so we are happy with how it’s playing out. I think it’s very early days too.
What's described is very much like what a sysadmin was in the 90's, prior to more specialized roles.
everything in this article is what I do as a sysadmin.
part of being a sysadmin is practicing devops. not the other way around
You're totally right in that sysadmins can certainly practice (part of) devops, but anywhere I've ever gone it's less the rule then the exception and "sysadmin" is a career-limiting classification, akin to calling oneself a programmer [0]. Ends, not means, etc.
[0] - https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-pr...
I've been obscenely compensated for working on interesting projects for much of my career. then again, part of it may be due to the confidence that comes with focusing on my work instead of fluffing my title.
Today, if I had a notion to, I could probably call myself a "sysadmin" and not have trouble finding well-compensated work. But when I was freelancing, as well as when I was earlier in my career, I can tell you a lot of doors would have slammed right shut had I used the term regardless of my capacity. I do not particularly love the term "devops", but at this point its umbrella at least encompasses what we (like, you and I, not the general we) think about when we're doing this stuff, and it has a certain amount of loaned--or stolen--credibility.
As I said in a top-level comment, I'm more twitchy at this description of being a consultant than I am at the devops part.
A better name might just be modern-sysadmin or sysadmin v2.
My role here is arguably a more integrated one where we're working towards a devops-flavored culture, and the infrastructure team here is part of it: "we maintain the roads and set up the water mains and keep the power lines from falling down, but hooking up your house to power and electric and making sure it doesn't catch on fire--that's your responsibility."
Best I can tell, like Agile Methodology, Skype, and open offices, "DevOps" is edgy jargon the financial people came up with to screw us knowledge workers.
I imagine myself trailing a parade, wearing oversized shoes, a rainbow fro, painted on happy face, and red nose, pushing a wheel barrow, scooping up the messes left by horses and elephants.
Some orgs seem to have figured out how to turn lemons into lemonade. That dude from Gilt has a "Test Into Prod" thesis that seems to square the circle. (I can't say from personal experience, but someday hope to work somewhere willing to try it.)
The majority of my time is either wrangling AWS, GCP or in tools such as Packer, Terraform, Ansible, Docker, Kubernetes, and writing bash scripts. I prefer DevOps now to writing traditional code as it is less down in the trenches nitty gritty code/math/logic and more higher level architecture and applying changes. I think it lends itself toward a slightly different personality type than a software developer.
> knowing a triple A (Authorization, Authentication and Access Management) solution
AAA is Authentication, Authorization and Accounting.
edit: formatting
The complexity and problem solving of the job is not automating some widget or service. That is very easy and not a challenge to someone with a deep programming background. The challenge is knowing what to build to a) enable developer productivity and flexibility to b) increase development speed to c) provide products/services the business side can iterate on quickly while d) not completely destroying the infrastructure and lessons learned that already exist and e) providing training and leadership to system admins and developers not used to these processes.
https://smcleod.net/tech/2019/08/08/camels-and-unicorns.html
I consider my duty as 50% as consultant regarding DevOps culture and agile culture and to 50% "building pipelines and all that automation stuff" This shows quite clearly how DevOps as a term is used in Germany. Either the fusion of responsibility of Dev and Ops is meant or just someone who does this modern cloudy stuff One thing I really like in the DevOps space is that I never had a problem with the tech stack at a customer. Some want more cloud, some more pipelines, some more infrastructure, still I could do every project and never had problems with my qualification. So when I have calls from recruite I kinda skip the tech stuff, since it won't be the issue anyway. I'm more intersted in how much the company really does DevOps and Agile. I just recently declined a job opportunity bc the company wanted to buy this modern DevOps stuff but actually couldn't risk an actual change.
The customer I currently working for is a mess. It's a middle sized company who has to finish 3 projects at the same time and do neither the "new world" nor the "old world" They overthrown a lot of the old world rules and took a lot of power from the project managers. But what they are doing now is taking random parts of the new world and implement it in a broken way and say "we are modern!" Like this they are doing neither of it and they end up with basically just chaos. Companies often want a change without having a change. And for me it's kinda annoying to run against corporate walls, when I was hired (as an expensive external!) to solve those problems but then getting ignored. So they throw more people at the problem and just churn them. As you guys already know, what 1 guy in IT can do within 1 day, 2 guys can do it in 2 days.
P.S.: I'm planning to relocate from germany to NY. If someone needs a DevOps, give me a shoutout to: jjdjnrjfifj@gmail.com Also open for non-work contacts. Don't want to be alone in NY
However, in my experience, I find that projects are incredibly hard to acquire, especially if it's your first freelance gig.
Do you happen to have any advice for me on how to get into DevOps freelancing, particularly how to acquire projects?
Thanks
That’s DevOps.
The single-most useful skill I have used is problem solving in an unknown context. Knowledge of architecture patterns help a lot in both cases. There have been times when I have had to read code just to figure things out.
There was a technical interview for a devops role I did recently. Whomever designed it was brilliant: a deceptively simple task (install an obscure blogging platform and get it running) with many hidden landmines and gotchas, compressed into 45 mins. Just another day at work.
DevOps engineers are thought as limited to CI/CD or infrastructure management. I would 100% consider them to be able to pick up development and understand the full stack (including networking, database management, etc.).
Kudos to you and others that expand "DevOps" to be more than just a simple shift of SysAdmins to Cloud.
It gives just the right characterization of qualification - "I've used it and been paid. I'm not necessarily an expert" - needed when living in a rapidly churning set of technology ecosystems.
I’m being facetious. Sysadmins always wrote scripts and developers always got called at home.
Many of the rest did, but not high quality scripts, and only in shell; not perl or python or tcl or ...
The lower rungs of sysadmin were happy to point and click on guis and lug and plug hardware.
It makes it more manageable to juggle the diverse work we are responsible for in a "DevOps" role.