School of SRE: Curriculum for onboarding non-traditional hires and new grads | Hacker News Reader

School of SRE: Curriculum for onboarding non-traditional hires and new grads

github.com

177 pointsgautamsomani60 comments

SRE has all these grand ideas like you need to know syscalls, debugging with strace, linux internals, designing major systems. Then you get on the job and find that your writing yaml and json all day and the interesting and impactful problems all get passed off to swe. Then you just get a check list because SWES making a quarter of a million dollars don't know how to log, add in metrics or use a DB apparently.

How to actually be an SRE, bullshit at a high level, work on a redhat cert, don't implement anything of value, bikeshed about issues with the people actually getting work done and pretend that writing yaml in git is "coding". What a demoralizing career.

This is the result of a poor org structure and poor leadership, not something particular to SRE.

You can take a look at any sre job description that gets posted. And I'm interviewing for these jobs right now. I talk to people constantly about what they do day to day.

As an SRE for 9 years: that’s nowhere near my experience. Depending on the period, the majority of my typing is either C++ or documentation. And the majority of impact is consulting on designing complex systems.

So youre a swe who does maintenence work...

Every sre job is going to quiz you on stuff like terraform knowledge, and then expect run their ci tools and manage aws.

When was the last time you changed jobs?

9 years ago. Got hired as a software engineer with a mission to engineer reliability into the systems. The definition of SRE.

That’s because you work at google. What tp described is sre experience at every other shop out there

This just means everyone else just got it wrong.

The market dictates what things mean, whether or not we agree with it. Is DevOps a position, a philosophy, a framework? According to the job market it can be any of things. In my experience interviewing for and working in SRE roles, most people are doing exactly what other people are saying in this discussion. A lot of SRE job descriptions are looking for people who do IaC to manage their cloud, handle CICD pipelines, and set up monitoring and alerting, all of which is ultimately a bunch of YAML (or JSON) with a bit of Terraform and Bash or Python.

Anyways, the point is it doesn't matter who is "right" or "wrong" and really we should be treating this role what it's being hired for, and as such education and training should reflect that.

Or that everyone else doesn’t have google’s unique technology and scale challenges that require anything beyond rebranded sysadmins. You decide

IME, it's much of the slog you mentioned, until things go sideways. Then suddenly, the few people in the org who actually understand the nitty-gritty are called upon to fix things. Rinse and repeat. That said, at my last place the SRE team did actually create some interesting and useful tooling.

> SWEs don't know how to use a DB

If by "use" you mean access it through an ORM, they do. If you mean "have an inkling about normalization, index design, and the query planner" then no, they absolutely do not.

Am working in Data Engineering and get the exactly same treatment:

Have to use some half-assed YAML based solution that compiled into Airflow DAGs. I'm basically a YAML "engineer" wondering what should I write in my resume: "I implemented hundreds of DAGs in YAML", while the other team bags all the credits and pat on back because they made a "great data platform".

BTW how does one get into a real SRE job? Or maybe I should just realize I'm not good enough for most of the "better" jobs out there, and should stick with whatever I have and hop once 4-5 years until retirement.

You are just describing a junior position (more or less).

Do you understand the business and can contribute to it? As SRE you can help save cost in production (harder to make money directly by building products or features). That's a way to a more senior position.

Can you automate the task you were given? can you use you new free time to identify where else you can save cost?

My conception of the position is that you can throw whatever data at me, but as long as you tell me the requirements I can get the data through with minimum cost and maximum accuracy.

I used to think knowing the business is important, but now I understand that it's impossible to know every business I'm in. I should rely on the BI for requirements and just do my job and do very well. I don't want to be a product manager or a manager so I prefer to stay in the technical side.

About the other questions, yes I automate as much as possible. I don't have any free time sadly as I'm just one man team (previously we have 6). I'm overwhelmed by this YAML engineering.

Anyway I'm starting interviewing soon.

TO add a bit, not every junior position work on YAML engineering.

mixing paints, crafting your own brushes, and studying the genetic geometries of the human body Will turn some painters into artists.

As somebody else pointed out in this thread, The trick is to refine your training in real situations as quickly as possible. The SRE Padawan isn't there to watch, they are there to grill me as to why certain things aren't in my documentation.

they are there to help me eventually raise up the operational excellence of our team.

With the jr-sre by my side, Refuse to perform a task in a vacuum, or jump into meeting alone....their success is our victory (and probably my vacation.)

I think that this is a great starting point, but there is some fun nuances to teaching how to survive a shitstorm that are so rarely brought up. Communication protocols, database warming, orders of operations.

you can teach these things but until you get them wrong you risk not learning them.

TLDR: good but missing a Kobayashi Maru.

This seems like a very good and necessary thing.

In general, the "onboarding" I have seen for non-trad hires and new grads at several companies has been absolutely non-existent... they pair these newbies with senior devs who either spend a lot of hours per week pairing and mentoring, or they ignore their mentees who then typically flounder.

It's an enormous wasted opportunity. A proper onboarding curriculum has a tremendous return on investment. While perhaps impossible to measure, you can't tell me that it's not easily 10x or something like that.

I am more of a traditional SWE, not an SRE, so I can't comment on this one directly. But in general, more of this.

One of the things that helped me out going head-long into a platform engineering team right after graduation was a map of our deployment platform's infrastructure and a self-paced guide on deploying an app from A to Z.

The more experienced engineers helped me with clarifying questions as they came up, but were otherwise not busied-out teaching the newbie.

Oh, nice. I love that. That really should be the norm!

Two things here.

1. As an SRE, it becomes even more important to get the big picture of all things that make the a company’s systems and services tick. I’ve been surprised by the lack of high level block diagrams explaining from this perspective. The diagrams are often from the perspective of devs as they build the nitty gritty details. It can be quite daunting for a new SRE, even more so for a new grad. On the other hand, I usually spend my first month documenting everything I learn in the first month as an SRE, and it becomes the guide that gets passed on to future SREs.

2. The thought of preparing educative materials to onboard a new grad(vs an experienced engineer who is expected to only match their experience to the company specifics) is often missing even in highly organised and functional shops. Nurturing and helping a new grad into the company specific things is a lot more rewarding and advantageous in the long run. There are almost always these pockets of tribal knowledge that exists among the senior folks who were there when some problems were solved. It is counter productive to the team as every time someone touches those sections, they seek the review/approval from those that possess that tribal knowledge.

I wish I had these when I first started:

The Missing Semester of Your CS Education https://missing.csail.mit.edu/

Check the playlists that say fundamentals https://www.youtube.com/@LearnCantrill/playlists

I know this has changed over time, but MIT's CS curriculum used to be really heavy on theory and light on industry-relevant practice (of course, students can choose a lot of their own courses past the core). For example, the intro course was taught in Scheme of all languages.

Thanks for the tip about that Missing Semester course.

I think MIT students are expected to have very strong self learning abilities and do a lot of experiments on their own.

https://www.quora.com/Who-produces-better-computer-engineers...

That’s what summer internships are for. Academia will be poor at teaching this anyway

> or they ignore their mentees who then typically flounder

This isn't surprising given the common strategy of replacing senior SWEs with junior ones (who are simply relabelled "senior" after a few years training).

Or shorting senior SWEs by giving them mentoring tasks, while keeping their normal duties - and evaluating them (wrt advancement) primarily on their normal duties.

SRE isn't a technical role. I mean, unfortunately we Ops folk are commonly given way too many responsibilities, so it's common to be technical. But if you read the SRE books, it's actually describing business management processes, product ownership, project management, NOC, monitoring, incident management, etc. Nowhere does it say "to be a good SRE you need to know how to use tcpdump".

In addition, Operations is much more of a traditional trade than a technical skillset. You can't read a book about being an electrician and then expect to do a good job wiring up a 100 amp panel and not killing somebody. You have to be an apprentice first, then after a few years a journeyman, and after a decade or more a master. Until you work under someone and have years of experience, you're not going to be very good.

This LinkedIn curriculum is just one example of many companies that misunderstand and underappreciate the true nature and purpose of the role, and make a sad attempt to cut corners by hoping they can turn a total newbie into a journeyman by learning some Linux commands. Doesn't work that way. (But thanks, LinkedIn, for telling me that I don't want to work there)

You just described an engineering curriculum. Understanding your technical field at depth and knowing things like project management, management, business, finances etc.

I dont understand this point, Almost all (really all) SRE job postings ask for the skills covered in the posted github link here.

I can see the need for project management, ownership and processes being needed on top of the previously mentioned skills, but skills in the link are exactly what most SRE jobs ask for.

> Almost all (really all) SRE job postings ask for the skills covered

Hiring managers are morons. They don't know what SRE or DevOps actually is. They think it's a systems administrator who can write Python.

These guides are so basic it's remedial for anyone with experience in this role, but highlights the problems of finding someone on the market that is exceptional in such a multitude of skillsets.

I feel like most companies eventually hit the "it's hard and expensive to find experienced engineers, let's hire cheap code school grads or other nontraditional hires en masse" phase.

That's not a good sign IME[1] but it seems inevitable, so it's crucial to make the best of it - I think standard intro curricula are a great way to do it.

It's also a pretty clear "canary in the coalmine" indicator. If a "nontraditional hire" cannot master or simply can't be bothered to learn the onboarding curriculum, that's a clear warning sign w.r.t. the viability of that hire.

______

[1] To be clear I'm not entirely against inexperienced/nontraditional hires. I'm against the situation where a company wants to make the absolute newbies a majority of their hires because like, hey, they can learn from the senior devs by osmosis or whatever.

Eh, not necessarily. While the internal SRE education at Google starts with assuming a slightly higher skill floor, we make every new hire go through it. Including people with a relevant PhD and/or a decade of experience in similar roles. Just making sure everyone is on the same page regarding jargon is probably already worth it.

Even funnier, my extended team requires further education in the particular skills we need here. Even an internal transfer with years of Google SRE experience needs to go through it. It’s just cheaper to spend some months on probably repeated training, than to discover a knowledge gap/misaligned intuitions in the form of a large post-mortem.

Hate to say it but that was my first thought as well. Read the section on signals. Besides being wildly incomplete and out of context (and basically useless in practice or in theory for those reasons), it's stuck in a section titled Linux Advanced, which I think says just about everything that needs to be said about the rest of the content leading up to it. Also, it's fucked in the head to put containers ahead of signals in your curriculum and it smells of YAML.

I don't know what LinkedIn is doing and how many SREs they've got sitting around "writing SLAs", but I'm not sure you should even be passing an interview for an internship if your operating system fundamentals are that weak. In anything anywhere near a production Unix system.

Perhaps I just don't understand what the actual role of an SRE is. I'm fully aware of what an SLA is. What I suppose is confusing is how the role of SRE came out of nowhere fairly recently. I know what a systems administrator does, and what a systems programmer does, and of various forms of title inflation that can be applied to both. SRE remains a mystery to me that crawled out of the web title soup sometime in the last decade.

Is it operations? Is it writing contracts and badgering programmers about that time they broke production? Are they like auditors? Do I need to take the SRE out to dinner and butter them up? I'm not quite sure based off of this content.

I've got nothing against nontraditional hires. I never finished college. But I absolutely despise the idea of somebody educated in this way being put in a position of authority.

I have heard this role thrown around a lot more recently in our progression to cloud deployments. I still haven’t decided if this is just an attempt by companies to get even more work out of a single employee, or this new combined role is really a necessity.

Personally, I think this is one area we are lucky to have existing certs: LPIC. Just sign up to Linux Academy (now A Cloud Guru) and do LPIC 101, and you have a basic understanding.

That's not onboarding, it's training.

SRE onboarding is about showing how everything connects, SLAs, lifecycle management, etc. Essentially, anything you wouldn't put in a public repository.

This is a good idea, but I do have some issues:

RDBMS:

> A subquery is generally a smaller resultset that can be used to power a select query in many ways. It can be used in a ‘where’ condition, can be used in place of join mostly where a join could be an overkill.

In nearly all cases, a join will out-perform a subquery. If you don't need the joined table in the result, a semi-join is usually a good idea.

Unrelated, in discussing MySQL configs, they link to the docs for 5.7, not 8.x.

> Adaptive hash index: Supplements InnoDB’s B-Tree indexes with fast hash lookup tables like a cache. Slight performance penalty for misses, also adds maintenance overhead of updating it.

IME, on high-traffic DBs the hit ratio is so low (not to mention the mutex locks) that it's faster to just disable the AHI.

> [Joins are] powerful but also resource-intensive and makes scaling databases difficult. This is the cause of many slow performing queries when run at scale, and the solution is almost always to find ways to reduce the joins.

This seems to imply that the answer to slow queries involving joins is to denormalize, which while sometimes valid, is definitely not the preferred solution. If you have a slow query, odds are high that something isn't properly indexed, or your schema design is poor to begin with.

Bash scripts:

* Inconsistently quoting variables

* Exit status 2

* Seemingly being unaware of else if

* Backticks instead of $()

* Little to no discussion on pipes, and the vagaries they introduce

This looks to be pretty useful material for beginners. But I must admit that “Exiting the VIM editor” requiring its own section made me chuckle.

> Site Reliability Engineers (SREs) sits at the intersection of software engineering and systems engineering

Call me a purist, but there is nothing in SRE that relates to what I consider the engineering discipline of Systems Engineering e.g. https://www.nasa.gov/sites/default/files/atoms/files/nasa_sy... & https://sebokwiki.org/wiki/Guide_to_the_Systems_Engineering_...

This sounds more like the intersection of software development, applications, and operating systems

You emitted your opinion but could you elaborate on why you think so?

We're very casual about throwing out the term "engineering", and maybe this came from the deceptive way in which Microsoft came up with the Microsoft Certified Systems Engineer certification which then entered the lexicon, or the use of the term operating systems.

True Engineering is not software development, and vice versa. The knowledge required to be an excellent engineer is not the same knowledge required to be an excellent software developer, and vice versa.

Even if we look at the most software centric parts of engineering in the fundamentals exam for the PE, we see a huge emphasis on the physical world that is key for Electronics Engineers, but not for a software developer https://ncees.org/wp-content/uploads/2022/09/FE-Electrical-a...

> onboarding our entry-level talents into the SRE role

This hr-speak is terrible. “Entry-level talents”, omg.

What would you use instead?

Newcomers? Juniors? new hires?

Newcomers and new hires both simply mean new to a particular job. There could be a newcomer SRE with 15 years experience as an SRE.

Juniors usually implies generally no experience with anything. For example, I wouldn't call someone a junior if they had 15 years of Windows UI programming experience but for some reason decided to switch to a Linux / SRE careeer. But they would be entry level in their new field.

Does anyone know how much time is allocated to this curriculum (asking as I do something equivalent for onboarding new developers)?

I would expect people applying for these jobs to know everything covered by this. That said, even the authors made some mistakes with this.

https://linkedin.github.io/school-of-sre/level102/containeri...

>Container engine is the intermediary between containers and Host OS

This is wrong. Both the containers and the "container engine" runs on the host OS.

https://linkedin.github.io/school-of-sre/level102/linux_inte...

>Below are the most popularly used file systems:

>ext

I have never in my life seen ext used. ZFS, XFS, and BTRFS are way more popular.

>https://linkedin.github.io/school-of-sre/level101/databases_...

None of the advantages of NoSQL given are true advantages over SQL. They are mostly just based off the database itself and not whether it that database is SQL or not.

>NoSQL systems tend to map data based on the programming data structures

Surely the authors must know ORMs exist for SQL databases too.

> I have never in my life seen ext used. ZFS, XFS, and BTRFS are way more popular.

I seriously doubt that any of those are more popular (ZFS is out of tree, BTRFS is starting to get traction but only just, XFS I'll grant in the RPM distros but it's mixed outside of that), and also I'm curious how you managed to get very far without seeing ext2 or ext4 at all; most cloud instances I've seen use ext4 for root, and some form of ext is popular for /boot anywhere that isn't merging /boot with the EFI partition.

The page says that FAT, NTFS, ext, ext4, HFS, HFS+, and NFS are the most popularly used file systems. I have seen plenty of ext4, but no ext. ext was removed from Linux before I was even born.

In regards to ext2 and ext3 I have seen them, but not in the context of being on a server.

Oh, the youth.

> I have never in my life seen ext used. ZFS, XFS, and BTRFS are way more popular.

Kubernetes just default to ext4 over xfs. https://kubernetes.io/docs/concepts/storage/storage-classes/

ext4 is not ext

maybe there's a typo there and they forgot to type the number 2 or 3. would you be satisfied if that were fixed?

They didn't put it in the description for it either. I doubt it's a type.

typo

> I would expect people applying for these jobs to know everything covered by this.

Don't take this personally, but this is super naive.

I have over quarter of century of experience and I am far, far from knowing everything. And even if I do, it is still useful to sometimes find materials that present the information in an organised way.

When we talk about people applying for jobs... you would be surprised. Majority candidates don't know more than a small portion of it. The best you hope is they are good at couple things and then know something about a lot of others.

I would say, looking at this site, if a person actually knows, understands and had experience with most of these things in their career (without using resources like this site), they are likely already well above SRE level.

What’s above SRE? You mean Senior or higher or managing/tech lead?

There is a lot of things to if you have good grasp of technology a company runs on and you are intelligent and able to work on some complementary abilities. Which is what you should be doing -- there are diminishing returns in learning more technology, but you get huge returns from learning completely new skills that are complementary.

Most of those things rely on some additional management/leadership skills. Yes, manager, tech lead, architect, startup co-founder etc. That's natural, because that's how you get more leverage to have more impact/return for your skills.

But that's not the only way to make more money. If you don't like management path you could, for example, become highly paid consultant.

I think ext could be interpreted as 'ext*' meaning ext{2,3,4} .

Then why would they then follow ext with ext4?

Reply on news.ycombinator.com