I am genuinely curious why this is the case, am I missing something here?
Even my local grocer's website somehow needs Kubernetes skills all of a sudden, not kidding at all.
Google didn't necessarily invent it inasmuch as they coined a term for keeping systems running.
I believe distinguishing the engineering from the engineer in SRE is important. The engineering side is essential but a dedicated full-time engineer may not be crucial.
The amount of engineering required depends primarily on the delta between your required availability (captured in SLOs) and your current availability.
The higher your availability needs are the more engineering effort you'll need as the code travels from commit to production because availability doesn't start and end in production systems.
The framework exposes or makes explicit considerations to achieve that goal. Oftentimes it comes down to a choice in balancing the cost between human time (or manual effort) and automation to replace it,and out of that the technology to achieve it. You may very well find that your availability requirements are so low that you can get by with a human pressing a button every few hours, and that this very manual approach is acceptable.
Going through the process will answer most of the questions on scale, whether a dedicated Site Reliability Engineer is required and so on. The SRE framework scales even if the system themselves don't require sizable scale.
My take away is the switch to focussing on reliability from a customer perspective, infrastructure failure is inevitable, impact to the end user doesn't have to be.
To understand how any failures do impact your users, it's usually very important to pick some good metrics and make informed decisions on what level of reliability you want and how you are doing.
That then feeds decisions on where you may want to improve your own tooling/processes to improve these metrics/end user reliability.
The dev focus is because you need the skills to automate and fill gaps in your tooling and you need to be able to understand the software you want to measure.
I think premature optimization in uptime is common. Yes, being down a bit is costly, but may still be cheaper than a development team being obsessed about adding 9-s to the service level %.
OTOH, obviously for Google, downtime must be minimized at almost any cost.
It is all about scale; but scale of budget, not traffic.
This is discussed pretty explicitly in the SRE book, in particular the idea of an error budget. Obviously they use some google services as examples, but the key message I got from it was "think about how much downtime is acceptable to you and work around that".
> OTOH, obviously for Google, downtime must be minimized at almost any cost.
Interestingly the SRE book pretty explicitly says that this is a poor goal for pretty much everyone. The cost of chasing more 9s goes up exponentially, while for most users whether your service is 99.999% available or 99.9999% available makes no difference, because 0.1% of the time their shitty router crashes and they have to restart it.
Better to pick a level of availability that strikes a balance between cost and user experience then work towards that.
1. define some reliability target (better expressed by some SLOs) in advance and what steps to do if that is not reached 2. if the service fails to reach it, do the steps to increase reliability arranged in step 1. 3. repeat at some regular intervals
The point I think is that the things are arranged in advance. Not after some shit happens because people get very subjective about "their own" service. The target is there, so lets try to reach it. We have error budget as well, lets use that one. If you don't have anything (as I've seen in a lot of places, or wishful 100% reliability), you'll have major reliability problems I'm absolutely sure.
So the SRE book tries to give you a solution to a lot of headaches some medium to large companies might be facing.
Therefore, while I think knowledge of SRE is important, at the same time you need to fear cargo cult thinking.
A (imo) worse trend I'm seeing is the focus on microservices. I'm in the Golang slack channel, and on a fairly regular basis a fairly inexperienced dev comes in and asks about microservices - implying they're solo developers that need to set up a microservices architecture. The cargo cult is strong, and people (in general) underestimate the amount of work involved in setting up a proper distributed system (microservices on kubernetes with SRE or otherwise). And people underestimate just how much software and engineers a company like Google has and maintains.
I mean my biggest software project(s) had at best three dozen engineers. Those + additional staff already filled a building. Someone decided that we should do microservices, but even then I thought it was overkill, it's amounts of traffic we WISH we had. And of course the split was wrong, it was a rough split on domains instead of something more solid (e.g. we had a separate 'address book' service, and an 'order history' service that was separate from the actual 'order' service. Which was a gateway to an older Java application, which itself was a gateway to an even older mainframe. I mean those two needed the attention, the microservices (which were written in Scala because otherwise they would be too boring) were just self-gratification.
I have no clue if they still have them. Part of me is thinking of going back there (I was working there as a contractor at the time).
Maybe I’m hurting myself, maybe I’m helping, who knows, but it’s become a recent tactic of mine in job interviews when a company demands Kubernetes skills (which isn’t to say I don’t have them, I’m just not as effective as I’d like to be...YET) to look dead in the camera and ask:
“How many nodes are in your production cluster right now, how many are control plane nodes and when was the last time two of them failed?”
Just to see if they’re putting in Kubernetes because they actually need it, or if they’re just faffing about with new tools because new tools and are struggling to even keep the lights on for their 3 microservices.
If there’s a better question to reveal this, I’ve got an open mind about deploying it to production.
What is a good answer and a bad answer for this, and why do two control plane nodes have to fail?
But a good answer would probably involve articulating some kind of genuine need that can't be solved with something else, I'd hope the answer involves some candor that the company looked at their options, vetted out that K8s was what they needed, why K8s is going to/has already solve(d) them over something (anything) else (nomad, or others but not necessarily limited to nomad), and if the interviewer is so inclined, disclosures of what they've learned trying to bring k8s into the house.
This last bit may be a follow up question from me "what have you learned so far implementing k8s? What would you have done differently if you knew better at the time?" etc.
What I'm looking for is a determination if the team thinks through the problem deliberately or if they're simply throwing their stuff at kubernetes because someone who no longer works at the company sold the leaders on kubernetes and no one asked any questions as the org jumped into those waters.
Mind you, this part is important: I am NOT anti-kubernetes, I am anti-foisting-kubernetes-on-platforms-just-because-its-hot-and-exciting. If a team wants to R&D it somewhere, prove out it's merit, create an MVP internally and can show the effort is worth undertaking to redeploy services on k8s, awesome.
My lived experience has been more teams than not are doing it because a thought leader somewhere wanted to pad their resumes, and then left the company, now the rest of Ops is paying down technical debt they never took out loans for.
However what’s happening is that the way we deploy software is being automated and standardized.
The dream scenario we’re trying to achieve is to tell the system here’s my application here are are some servers now go deploy it and keep it up and running.
This actually does work even in the face of failures. It’s just complex because it’s new and solving a very hard problem.
I would considers these systems as a form of AI.
When you manage tens of thousands of servers these investments make sense. And once these systems stabilize they will be usable without much difficulty. Which means in five years you’ll have an in demand skill if you learn it now.
The thing that caught my eye is infrastructure-as-code. It's massively useful to have your setup in version control, regardless of how big you are. A few years ago I was managing code separately to eg network configs, typing commands into routers, that kind of thing. It's a natural step to have all the infra in a form where you can see changes as well as reliably deploy.
I thought that's what DevOps was?
My understanding - from one conversation with an SRE one time over a beer! - was that an SRE's job is not to keep things running and scaling, but to work with development teams to help them keep their things running and scaling. So the key idea is that SREs get leverage by having all other developers work on operability.
SRE is the same. DevOps, microservices fall in the same bucket.
The irony of course is big companies, the ones that really need DevOps, microservices architecture etc sometimes have the hardest time to adopt new ideas to leverage them. Case in point, I work in a large FinTech company, we are talking multi-billions. 2015 small group wanted DevOps, I became that, only now, 5 years later the greater part of the company is starting to adopt it. I'm still giving presentations on VM vs Docker differences. While at the same time the team that wanted DevOps deploys to production Openshift clusters multiple times a day.
"SRE" may be a Google-initiated label, the practices behind predate it (first sight I've seen was Flickr in 2008).
> A cargo cult is a millenarian belief system in which adherents perform rituals which they believe will cause a more technologically advanced society to deliver goods. These cults were first described in Melanesia in the wake of contact with allied military forces during the Second World War.
Personally, I'm still trying to wrap my mind around the implied context here.
Sorry it's slightly off topic - but perhaps someone can clarify?
Thanks :)
> In the South Seas there is a Cargo Cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land.
It's about copying others to achieve a goal — but because you aren't copying the motives, only the visible methods, you don't succeed. In this case, it's about copying the engineering practices of big organisations without being a big organisation yourself, in the hope that you become one. Hope this helps.
I hope this is a valid example?
1. Project P fails spectacularly, really bringing the reliability targets of the whole production down. In reality we don't have proper targets, we have "100% reliability" target, but still, it was below 98.5% judging by my calculations which was unacceptable.
2. Management furious at that time. The team in charge of that project promises that they are going to regroup, start doing "retros", using "planning poker", and introduce more alerts (to some other team of course...) etc
3. Management gives them another (n-th) chance because the tools sound cool and think that employing "technology" will always fix social problems.
The truth is, noone knows we deployed the thing using caprover by just wrapping the app in a docker container.