DevOps, SRE, and Platform Engineering
iximiuz.com
iximiuz.com
- DevOps: Hired to do everything not involved in the feature development of main business. Range varies from Terraform to maintaining Jira, GSuite, Jenkins, ETL, BizOps. Maintains scripts and hacks everywhere.
- SRE: Orders teams to define SLOs, focuses only after incident procedures, not before. Insists on creating dashboards and putting them on TVs (when we had offices). Most eventually become vendor fans.
- Platform: We made this, we know better than you, you cannot use anything else. If you need new thing/improvement, create a ticket we will get back to you in 6 months to say we cannot do it due to other company wide very important initatives.
I love not being a regular developer and work on platform side, but its a very irregular space and very hard to hire.
I think it's just genuinely too difficult for most companies to do well and it's too easy to fall into the same old well trodden traps.
Documentation is something everybody professes to keep up to date, clear and complete, for instance, but IME it never truly is.
I think one way of reading the comment above is that it's the effect of business incentives / Conway's law on various approaches to staffing the non-developer parts of your organization:
- DevOps is incentivized to be an extension of the dev team and get stuff into production, i.e., in the average company hiring "DevOps," the engineers operating production report up into existing dev management. Therefore you get "scripts and hacks" because you're never given the mandate to do anything better, and you're evaluated on how much you can manage to ship things to prod without building separate specialties, not how well you ship it to prod.
- SRE is incentivized to demonstrate good SRE practices and their value to the business. Incident response is a lot more visible than incident prevention. Picking an observabiity vendor who can get you nice graphs is a lot more visible than spending months making the right graphs. You're evaluated on how much you improve "reliability," which means you need to figure out how to measure it (and display it) in the first place, but it's not clear your measurement is what the broader business would measure.
- Platform engineering is incentivized to build up a well-defined platform and have their own products/services that are used internally. They are a development team of their own. So their value is most visible when those products exist and are widely used, and a little more cynically, when those products aren't unmodified vendor or OSS products. You're evaluated on the number of people using your product and how infrequently the rest of the company says "I'm just going to run this myself, give me an AWS account."
Any of these disciplines can be done well with strong management support, by which I mean when management actually cares about the long-term problem of running production well and can make informed decisions about how to staff that problem. (For myself, for what it's worth, I'm in a platform engineering organization but my current role is 80% to make it easier for people to use OSS and to reduce our deltas and 20% incident prevention.) But when incentives get disconnected from the business and the inertia is to keep delivering what people thought you should have delivered in the past, the disciplines break down in different ways.
Most developers just need something like heroku, tsuru, appengine, or openstack. But because kubefans want to have fun too and try to be as cool as Google or Facebook we end up having to deal with tons of custom Golang tooling with fancy names, half assed documentation and grumpy looks when you ask for help because you don't understand the 38 steps guide to deploy a service.
$100 is not really a price point for any of the things you mentioned, and the way PaaS charge is often restrictive of development model (want microservices? we charge by the service. want a monolith? sorry, we only have instances up to this size. need to scale in smaller increments than "one machine"? too bad.) When I helped build my first platform, it was a migration from Beanstalk to Kubernetes (1.8 at the time).
Not only did it save 70%+ costs, it allowed developers to cut services as large as they wanted, and scale them as fast or slow as they wanted. Metrics and logging were nothing they had to worry about on the infra layer either.
I wouldn't build these things again, OAM / KubeVela is a nice compromise if you want one.
Also, I wouldn't give a dev direct access to Openstack the same way I woudn't give them access to Kubernetes itself. They have too many footguns and too much complexity, either you end up with half-baked deployments or half the teams time spent on doing things the right way.
The typical case of feature teams not being allowed to refactor / reduce technical debt was extreme. Some frontend requests would result in hundreds of backend requests for no good reason. We've always been ignored by management, so cluster was on the verge of coming apart at the (mostly "spending half the cpu time in conntrack") seams for years. So close, it'd sometimes just die for a minute, probably because someone decided to use HTTP way too liberally (network is free, right?). I had a pcap file from a sample of 10 machines, and a 30 second sample was so large, it was impossible to analyze with any real-time tools.
No managed Kubernetes could match this out of the box. Adding a mesh to do more directed requests for this to be fixed failed even when we tried it. As far as I know they're still using the platform as is (in maintenance mode, with the contractor assisting) and fired that CEO pretty soon after.
Is this a bit of a special case? Sure! Does it still happen a lot, no matter your infrastructure? Count on it.
A company I worked for years ago moved our custom built platform to AppEngine Flex and it was the best thing we could have done. We embaraced DevOps as it should be done (not with "devops people" but with developers understanding their production environment). No need for contractors. Just the documentations and an easy to use, well tested and battle proof platform.
Similar experience many, many years ago with Tsuru, which is a kind of smaller self hosted heroku, although this was a company of about 30 enhineers
Kubernetes being "a platform for building platforms" still rings true to me, and it is extremely good at that.
One major issue was that the culture wasn't great. SREs without teeth and Platform Engineers that fix the shortcomings of development teams with glue more than anything.
As for why contractors, this was the IT division of a big retail business that refused to pay competitive salaries in solidarity with their other workers, but never fired anyone for underperforming (or not performing at all), so the people that actually get work done tend to leave pretty fast. They tried to compensate this with contractors. Some departments were fine, and I loved the team (high standards, created lots of value for teams), but it wasn't a great long term workplace.
A lot of companies don't operate at Google or Facebook scale but many companies operate at a scale which quickly outgrows the solutions you mentioned. Maybe you just haven't worked at places with larger scale to see the benefits things like Kubernetes provides?
Put into the equation all these salaries and the opportunity cost and you have a pretty juicy budget for appengine or similar.
The problem is complexity is variable. For developers who never really learn how their code actually executes, or haven't had to be responsible for tuning and such, this is great, but for those of us who want to know where the rubber meets the road, these tools are kid-gloves. How can we fully optimize and support our code if we can't see the compute on which it runs? What other "benefits" will the system bring that we have to account for?
> Then make it so well-documented and easy to use that teams don't need to be _forced_ to use the platform tooling, they'll just use it because it's the most convenient thing to do.
Very optimistic. In my experience with this kind of thing with homegrown platforms, documentation was a distant last, because it always is, and we weren't left with other choices anyway. I would love to pick the best tool because it is convenient, but that, too, is variable. Maybe I know Kubernetes inside and out - that's plenty convenient for me, then, and will be better documented and discussed than any homegrown platform because the world can contribute to it.
My personal experience agrees 100% with what he has said :)
I think it is due to cynical takes like this that the industry is evolving towards “Use whatever you want, so long as you are open to being on-call for it yourself. Or you can use this Platform maintained by other people.”
Nothing sucks as much as being woken up by pages because some new, shiny, and untested piece of technology was used in production and broke in an unknown way.
There is not an exact line. Platform team can easily abstract (i.e) Kubernetes deployment infrastructure, but not everything else that you might be needing.
As platform team member, you really have to know what to build, doing interviews, figuring out what developers and SREs really want, filter through what they are asking for, control through not-invented here syndrome, make build-vs-buy decision, create the community around the product you build within the organisation etc. You cannot just start building a PaaS (most common thing I have seen them build). It is actually quite complex if done right. Having said that, lot of organisation the definition is quite vague, political too and sometimes the boundaries are not clear.
The most common hurdle is simply that most developers buried in enterprise software dev have no familiarity with the cloud or foundational DevOps tech like infra as code. So we typically end up building some form of deployment mechanism cobbled together with packer and terraform, but the real value in bringing in outside help is that we handle training and change management necessary to actually drive adoption.
“Build it and they will come” is a terrible design philosophy. You can’t just stand up a platform team and expect it to go well: a more transformational mindset inclusive of org design and strategic outlook is required, and this type of DevOps / SRE transformation is no less of a radical change than HR or finance transformation programs.
Nobody asked for it and everyone is content rolling their own pipelines by hand, yet they've sunk a ton of time into it.
Originally it was supposed to be a SWE who could also do operations, but there seems to have been a quick race to the bottom to remove all the SWE tasks and turn them into pure operations policy wonks.
Having seen all of roles, being SRE was the only time when a slack message sounds would trigger anxiety in me.
> SREs tend to grow out of the software teams and are the people responsible for the pager, metrics, SLOs, hacking up turing computable YAML, memorizing the whole CNCF landscape, and yelling at Kubernetes.
Yeah, that's exactly what they were created for...
You didn't have to call me out like that.
Royce: "These are the flaws in the 'Waterfall model'" -> everybody looks at his diagram and uses it [1]
Agile: "Individuals and interactions over processes and tools" -> SAFe [2]
DevOps: "Assign development and operations the same business goals so they work together, rather than against each other" -> "DevOps was a cultural shift giving development teams more control over shipping code to production" (this article)
For example, agile was never going to work according to the rules, because "individuals and interactions over processes and tools" are not how most companies work.
There's probably a parallel to draw with programming languages. A few people will use Haskell or Lisp or Idris and be at the bleeding edge, but most people are satisfied with Java that sometimes cherrypicks good features.
Both of those seem to be rooted in aversion to risk, or (inclusive) managers making most of the decisions. I'd say that's because developers tend to have different values from other people, often valuing stability a bit less and "the right solution" a bit more.
You're hinting at what the original DevOps idea was trying to solve. The traditional view is that Developers value shipping features while Ops engineers value stability/uptime. It should be easy to see why these differing values cause friction between these two groups.
The way DevOps was trying to solve this was getting _management_ to assign incentives/values/goals that would apply to _ both_ groups. An example might be increasing user engagement.
Aligning these two groups should be the main takeaway. The reason why The Phoenix Project book was so successful was it perfectly described this adversarial relationship. I'd certainly seen the same thing playing out at several companies I'd worked for.
First, elite practitioners identify flaws in the standard practices of some corner of the industry and figure out a way to ameliorate them. They give it a name so they can talk about it, write about it, organize conferences etc. They're mainly interested in disseminating this good idea.
The idea becomes popular, and a cottage industry springs up. Some of the original proponents become consultants, but there are others who were early adopters, and many more who are jumping on the trend. You also get an ecosystem of open source and commercial tools. This starts to warp the original ideas, as they stop being about purely about improving results and incorporate a bias for generating more consulting or tool revenue.
This drives wider adoption. Companies start looking for people with experience in the new thing, and people start putting it on their resume. It gets applied in situations it wasn't meant for, sometimes on purpose and sometimes by accident. You start to see "no true Scotsman" discussions where detractors claim they tried the thing and it didn't work and proponents claim they weren't doing it right. There's also cargo culting, where people who don't really understand the ideas try to implement the practices with varying degrees of success.
Eventually things get watered down to the point where the terminology starts to just mean "good" or "cool". The actual practitioners stop using it because it's no longer useful for communication and it falls out of favour.
But. All is not lost! The original ideas were good, and they've been widely disseminated. They get taught to young people joining the industry, and they start to be part of "how we do things." They don't need special names to distinguish them from "mainstream" practices because they are the mainstream. In fact, young people sneer at them because duh, without knowing what they were a reaction to.
You can see this with Agile, Lean and DevOps. On the business side, "disruption". The cycles are slower and maybe less intense in more CS and pure coding domains, but you find it there too with scripting languages, object oriented languages, TDD. Functional programming is leaving the consulting phase and seeing wider adoption.
This is probably not how we'd like to think things work. But they do work.
I see each role as an evolution where you go from DevOps -> SRE -> Platform Engineer. Automating the infrastructure and tooling is the catalyst. You can have any of these roles in your company based on how far along the path your company is, how on board management is and how interested your development teams are in growing/learning these practices.
DevOps writes scripts/code to automate CI/CD pipelines, creation of infrastructure, deployments. Once automated you move onto..
SRE writes scripts/code monitoring, alerting, SLOs/SLAs, incident processes. Once automated you move onto..
Platform Engineering where you write the tooling to make all of the above codified. Tooling could be anything from cli tools to kubernetes operators.
I say this as someone that has spent the last 3 years automating my way towards platform engineering.
The job is what you and your colleagues make it out to be. You are there to fill a certain need and a knowledge gap within the team. Figure out what that is and you are golden.
I do agree though that the most important part of a professional life is to fill the gap within your team. But that's another discussion.
For some reason "DevOps Engineer" is much more polarizing than "FullStack Engineer" or "Senior Developer"
15k -> 30k -> 60k -> 120k
Also if you have stock at a successful company in the past couple of years things have gone vertical.
DevOps is great, but it's not a title. It's beautifully summarized in "The DevOps Handbook" and it's fictional counterpart "The Pheonix Project". It's about culture.
Some people might feel different and want to re-evaluate due to so many company adapting it in a similar vein many companies want to adapt agile. But make no mistake, you can't define it then, since then the context of the company defines the word. Thus sadly the word is slowly loosing its meaning.
This is likely due to lots of management people wanting to adapt the buzzword without understanding it. This since it has have had such a great success in many companies that actually know what it is, have taken the DevOps culture to heart and implemented processes in line with it.
Those who can manage to surpass that though can get great productivity from it though.
The analogy with agile is pretty good here. Sure you could have an Agile Engineer, that works primarily with making sure the agile process is actually used and help facilitating that. Normally that would not be a position itself though, especially not an engineering one.
Never had someone tell me I dont understand my own conclusion to my own article before. Thats a first.
If your plan is to just hire software engineers, you're going to have confused applicants, if your plan is to hire sysadmins, they probably will have a different skillset than you need. Most real companies hire DevOps engineers because that's the way to get the people with the skillsets you need.
Being so purist about this sounds to me like useless debate, but maybe you have better suggestions.
At my current place we stopped hiring DevOps and started hiring SREs + software engineers for specific internal teams, but for a few years there between 2010-2020, good luck getting people with DevOps skills without actually advertising DevOps roles.
What's DevOps skills in your opinion?
I asked for alternative suggestions and offered what we use. If you have suggestions of alternative titles you've successfully used to hire without slowing things down I'm all ears.
> What's DevOps skills in your opinion?
The most important skill is knowing to not fall for the bait that is this question so that you can actually work.
I think the only reason to advertise for it as a role, is when your tooling is too complex and you've given up looking for Software Engineers that could do it on the side. Which is fair enough, I guess, though providing a Golden Path nowadays is probably more popularly done by hiring Platform Engineers.
If you could write complex shell or perl scripts, and you knew how to dig into performance metrics on the fly, you were a god. (Yeah, capturing and displaying system metrics was in its infancy, and most shops simply didn't know how to do it)
While some systems were built to be "horizontally scalable", you were lucky if you worked with systems that weren't SPOFs. You were happy with redundancy via active/passive failover (and that was probably a manual affair). Maybe if you were really lucky, you worked for a SaaS that served a monolithic website via a fleet of webservers behind an F5 loadbalancer. "Wooooah!". But you still set up those F5 VIPs and pools manually.
The modern systems engineer rarely has to do anything like this, even though the underlying technology is similar. These days everything you care about is in git, and is mostly testable on git push. Modern companies, including tiny 4-person SaaS startups, can have infrastructure-as-code, configuration-as-code, packer configs, dockerfiles, container orchestration, CI/CD, metrics collection, USE/RED metric graphs for every system with alerting, pager, incident handling runbooks...
At some point over the last 25 years, the term Systems Engineer could mean that you're an expert in the old ways or the new ways. When you advertised for a DevOps engineer, it left no doubt that you were looking for the modern variety and expected that experience and skillset.
On another note, how amazing it is to define our systems so we can more systematically iterate on them and improve. I love it when developers get integrated into the process as well.
The general consensus where i work is that everyone does DevOps, you develop the app, you make sure you deliver the app, if there's any issues you debug. You ain't done until your ticket is in production, and you can't point to other to solve it for you, only help you.
Where does the DevOps engineer fit in there?
Improving pipelines. Dealing with downtime on the CI. Improving performance of the CDN. Writing better tooling for the devs to deliver value even faster.
If you are out there interviewing for a DevOps job and you hear things like this: run. Every meeting you have with this person will be a living hell. Endless debates on semantics and definitions while your project is due today.
I was incredibly frustrated with everyone I spoke to having a different definition of DevOps. For many it’s just sysadmins rebadged. For others they think they sysadmins couldn’t code so it’s an evolution. For yet more they think it’s feature developers who manage ops.
Nobody has a formal definition that fits, the term only exists to obfuscate responsibility.
I agree that Development should be mostly self-service and that the goal of any "Ops" person is to get out of the way. "Automate yourself out of a job" is a common sysadmin mantra.
However I don't agree with your other points, there's some over exaggeration when you say "if devops doesn't exist then thats a red flag", if devops can mean anything then it's a useless term and absence of it shouldn't be seen as a red flag.
>Shoud DevOps engineer be a role? Yes. Absolutely.
Small typo here, and of course I disagree with the statement and preceding paragraph.
I think "Platform Engineer" is closer to what your definition of DevOps is.
The point of my post is that everyone has their own definition, which is why I'm frustrated at the term.
> Infrastructure Engineer
Regarding this... This is actually my job title right now, but it exists outside of computing: "Infrastructure Engineer" being the kind of person who makes bridges and roads and so on.
Reminds me of an old old joke.
There are two kinds of Engineers. Mechanical Engineers who build weapons, and Civil Engineers who build targets.
But it was quickly coopted to be "ops, but cooler" because these sysadmins write code too. And then even that definition started to get coopted by people using it for other things. These days it can mean anything. Frankly, I see it as a small red flag. Anyone who uses it is basically jumping on yesterday's trendy jargon, and that's not a good sign.
EDIT: Great explanation from one of the commenters https://link.medium.com/tC5p4nfYCib
Industry has moved on and if you need somebody to manage direction of your infrastructure (yes, be it just Jenkins or whole AWS) you will search for DevOps Engineer. There is no serious tier.
If you needed dedicated people so dev teams can just throw it over the fence to them that already existed and was just called sysadmins/ops. No need to invent a new term for that. More like some companies didn’t get the memo that devops != sysadmin, not even a title at all.
> There is no serious tier.
Well who’s in a bubble really?
My conclusion is DevOps is a newly made up word. The technology community is still trying to agree on a definition. Like all words, that definition will change over time.
I think it is okay to have your favorite definition, but I would be careful inferring too much about someone’s “seriousness” or competence based on their choice of favorite definition.
I've held multiple titles doing this. Titles doesn't matter, because nobody respects what the titles implies.
Look for CloudOps, we often also have multiple certificates from Azure/AWS/DigitalOcean/Alibaba etc.
We not only keep infrastructure in code, but also make sure you are not ripped off by cloud provider and your architecture makes sense..
Maybe the next buzzword will be "platform engineer".
I’ve seen Platform Engineering take various different forms, but it’s current incarnation is split into 2 teams:
1. Developer Productivity. Ownership of quality, efficiency, and developer tools. A recent example project was improving the quality of our test data capabilities to make manual testing easier/more consistent.
2. Unified Client Platform. This team owns our design system and is the primary owner of our effort to unify on React.js/React Native, though a lot of the heavy lifting is distributed around the team.
(I mostly wanted to weigh in since at a mid-sized company at least, we’re not interested in building our own PaaS… k8s or Lambda is fine for that. Instead, we want to ensure developer onboarding is painless and anything specific to us is easily understood - the bulk of the surface area should be best-in-class 3rd party tools or ideally popular OSS frameworks.)
But I found a nice resource to reference where we have 7 "DevOps" team anti-patterns and 9 valid team patterns that people call "DevOps":
DevOps: Make the developers work with infrastructure better, deploy faster, have less errors!
SRE: Make the products more reliable, get failures down to the minimum, have less toil!
Platform: Be more efficient by standardizing technology across a large organization!
"Reality"
DevOps: A sysadmin that writes Terraform
SRE: A sysadmin that writes Terraform and knows when the site goes down and if it's important
Platform: A couple of sysadmins forcing everyone to use one set of technology in one way without ever talking to anyone about what to build or how to use it
I currently see myself in a position where I'm responsible for direction of a platform.
What best practices are there around common things like urn, config management, infra provisioning, and other common components.
I have my best practices, which I could enforce on users. But how do I know what is most important to focus on.
But with regards to purely technical stuff like what tools to choose and how to use them you’ll need experience, either your own or someone elses. There’s really no replacement in my opinion. I had loads of experienced people around me previously, but now that I work more or less alone in the field I rely a lot on community slack channels for the various tools and debugging. It’s amazing to be stuck trying to figure out how to best use Loki for logging, jump on their slack and get advice from the Lead Engineer working on the tool. Or getting input on what tools to use from 50 different people in different companies through /r/devops on reddit.
Our biggest focus, though, is providing the platform upon which our customers build their apps. We are a Multi-Tenant SaaS development platform. So we offer the APIs and run "The Platform" in production, which includes a database, a task system, a programming language, etc...
One of the biggest of those, and the reason it can be a challenging position to be in, is that the person managing the team, coming up with the vision and roadmap, and even the architecture direction are often the same person. Few people around you are experts in building a good vision, strategy, or roadmap for a platform organization. It’s a skill set heavily overlapping with product strategy but also completely divergent. If you’re in this position, you’ll be doing several of these things poorly at any given point in time. (My current employer sits on the opposite side and has made sure we have a fully engaged product partner to do this well, which is a breath of fresh air. It's a much different working relationship than with a product partner in user-facing teams)
It’s a high leverage position and can provide a center of excellence for the rest of the organization to emulate. You tend to be very visible among developers in the org if you’re doing it well, and you drive a lot of cross-cutting architectural goals.
On the team building side, hiring people that thrive doing platform work can also be tough! It’s lower visibility, but provides a unique set of technical challenges and an opportunity for building technical breadth you find in few other roles.
Early on in your career can be a great time to explore this sort of work. Repetition is low, and you build a lot of tech chops through the work. Try to be somewhere where it seems like a first class citizen in the organization ecosystem with engagement from product, too.
And for what it's worth, I don't think your general goal should be to attempt to build over buy. Helping teams adopt and successfully use paid-for tools is a big part of the gig.
Version control? Stack Overflow? The Internet?
I don't want to waste my life understanding your custom reinvented wheel Golang tooling around kubernetes and 50 steps guides to do anything.