Comment by dijit | Hacker News Reader

Comment by dijit | Hacker News Reader

What I found most fascinating is that Google essentially rediscovered what is important in a sysadmin and codified the contract between feature developers and reliability roles.

Instead of having feature developers feeling like they have no say in operational requirements, and instead of having reliability staff fighting unstable mess: properly making the contract means everyone gets heard.

Contrasted to devops which despite coming out later was in vogue when this book came out; which caused the muddying of the role of sysadmin to meaning either:

* a sysadmin practicing agile (the original definition btw)

* a software engineer with enough OS skills to carry the pager (the popular one), or

* a team consisting of sysadmins and software developers with no barrier between them (10+ deploys per day style).

Everyone had their own definition of DevOps. So when SRE came clear: sysadmins are needed, stop trying to push everything into one person, heres how we fix the tension between teams: it was a breath of fresh air.

The only revisionist history (that even google seems to forget) is that sysadmins could indeed write code, though it wasnt pretty and didnt have the nice things like mocks and tests. This has changed a little since 2010 at least but it is still dire, even with Cloud making things much easier.

*EDIT:* I've gone from +4 to 0 points in a very, very rapid amount of time. If I have offended you; how?

Sysadmin originally meant Unix greybeards who were competent at C and could write and implement, say, Kerberos. Then in the 90s-2000s the term came to primarily refer to Windows admins clicking around Active Directory and Group Policy for large enterprises. The "sysadmins can't code" thing came grom that time and the early 2010s when all the cool startups were building on Linux and the available pool of sysadmins were largely Windows specialists. Then DevOps came along trying to get Windows sysadmins that were dabbling in Linux into modern development practices, and SRE came along trying to revitalise the old Unix greybeard style with modern software development practices.

  * SRE are sysadmins with Go
  * DevOps are sysadmins with Python
  * Sysadmins are sysadmins with Perl

(stolen from twitter, can't find the source)

I’m a software engineer that also does operations work. But I really love Go for operations type tasks. Simple, direct, code that’s not too verbose, with predictable performance, and fast compilation to a single executable. I’ve even taken to baking an editor and the compiler into an image with a small library of code for operational tasks, that I can quickly edit and run in the deployed environment for ad how operational tasks.

Image?

Sorry, docker image. We deploy everything in Kubernetes, so I can spin up this image as a container in the cluster, open a shell on it, and use the Go tools we wrote for interacting with our services to debug and diagnose what’s going on.

I took their question to be shorthand for “Can you please share the image?”

Well no, if that’s the question. It’s the IP of the company I work for.

What if they have PowerShell?

That may what sysadmin meant back in the 80s and 90s but it's not what it means now. I would never describe myself, a SRE-SE, as a sysadmin because I would be describing someone whose primary job is to operate software and iterate on configuration.

SRE-SEs on the other hand are SWEs, they just have a focus in systems adjacent software whereas a SRE-SWE is someone who can dig into compiler level issues and optimization. Both write application code, do analysis, and write policy. A sysadmin of today would be out of place on a team like that.

I don't think that's how the SRE role is described in the book. It shifts the boundaries between sysadmins and developers to give developers much more freedoms in deciding the operational parameters of how their softwane is run/released but also gives SREs the ability to push back if they get handed over some piece of software that's not sufficiently following recommended patterns that make actually operating it mostly automated.

That's exactly right.

Can you please point to the part of my comment that you think disagrees with this?

I think I might have misclicked and responded to a different comment than I intended to. You were basically saying the same thing :)

i would think most people object to calling SREs sysadmins.

thats unfortunate, I think that means they arent aware that the job of SRE is functionally identical to a sysadmin from 2005 in terms of responsibilities and required knowledge.

We just live on a higher level of abstraction and have better tools & processes now.

Compared to 2005, we live in a society where a lot of people are very sensitive to words.

For example, "developer" is considered offensive, because for some people it's very important to be called "software engineer".

Really good developers don't care about titles.

They don't have time to worry about such, or they have so much money / experience that even if you call them "smart monkeys" they'll be happy with it.

Same goes with sysadmins, SREs, devops, or whatever role you choose.

For some people they have shitty jobs: they don't have such recognition (whether for a good reason or not).

No recognition from work, no recognition from colleagues, no recognition financially, etc, that, if you remove them the title / prestige, obviously they would feel bad.

Source: my experience in a school calling itself "engineering school", and all other schools calling it a "place where to pee code"

> Really good developers don't care about titles. They don't have time to worry about such, or they have so much money / experience that even if you call them "smart monkeys" they'll be happy with it.

That's about the money, not about being good at your work. Ask anyone on the street if you can call them a rat in exchange for a million dollar salary and they'll say yes. It's quite simple.

As someone who always knew I didn't really need to know code for what I wanted to do, I'd posit that a developer comes across as someone who may not be formally trained. Maybe they're a hacker, maybe they know a language or two and dabble. A software engineering is someone who is comfortable at various levels and understanding machine level code in so far as also being comfortable with software patterns.

-shrug-. That's what I feel like in my org at least.

I was a sysadmin (at uni, in the early 2000s) and I am an SRE today (at Google).

The two jobs are nothing alike, at all, whatsoever.

Sysadmins are support roles. Their functional role is to provide a healthy substrate to run the application layer on top of.

SREs work at the application layer itself. If the system can't scale due to internal architecture, an SRE would be expected to propose a new, scalable design. That would be in addition to maintaining the substrate.

To be clear, there is also nothing inferior about performing a support role. No org can succeed without support.

But the two roles are not the same, and if a job's set of responsibilities don't include shared ownership over application layer architecture, then it can be a great job but it's not an SRE role.

This seems unlikely to me solely from a growth perspective. That is, it is possible to reach "Principal" or "Distinguished" SRE as an IC, the equivalent of a director (or similarly: SRE Director). I don't recall companies having, or desiring to have large organizations of sysadmins.

I think one of the key differences to highlight is that usually SREs are engaged early in the design process of new features, and are often driving their own feature changes to the product for reliability or scalability reasons. Those aren't responsibilities or expectations that I've really ever seen in the context of a sysadmin.

One one hand there were definitely support-staff system administrators, but on the other hand there were people writing USENIX papers. Large system administration was system engineering. The people who wrote Nagios, RRDtool, bcfg3 and cfengine, and so on, were solving their own problems first.

I think people like Evi Nemeth, Tom Limoncelli, Æleen Frisch or David Blank-Edelman would have been the equivalent of Distinguished Sysadmins at the time. But they weren't at startups. The places that needed that level were universities, research facilities, telecommunications companies, and the like.

I was fortunate to work under Geoff Halprin early in my career, who while not as well known as those names was a SAGE and USENIX board member and who definitely planted the "don't just do task, engineer yourself out of a job" seed in me.

there's a sort of bifurcation of SRE responsibilities with one branch focusing on software-enabled automation and the other on "systems engineering" aka sysadmin. Both are called SREs at Google which seems to cause widespread confusion externally (and even within Google). Also see so called "Ben Treynor Curve"[0]

[0] - https://www.usenix.org/system/files/login/articles/login_jun...

Within Google: Treynor’s curve is a hiring concept. Once in, you’re doing literally the same job. Being in a team doing greenfield development it took me three years to notice that my TL, with whom we’ve been defining the mathematical model, designing and implementing the system, is on the SysEng ladder.

> Once in, you’re doing literally the same job

ok, but you don't tho (i'm sure there are exceptions). if you do, you will have issues with promo (ask me how i know) so if you don't care about that then sure. also if you're se, you had to interview to transfer out of sre to a pure swe role. i personally didn't have this particular problem but saw a few folks that did.

This might be an age-related perception. I think if you're over 40, you'd consider this complementary to SREs. The role of sysadmin, as it existed in 2000, is almost unimaginable now.

As someone that has had those titles for many years in the past: I'm only going to object to the one that's paying less at the time.

A good sysadmin was always doing the same thing as a good SRE.

I think you’re right, but I want to relate a story where the agile team leader decided that devops meant “developer operations”, or in other words the development team.

Sometimes all you can do is shake your head.

Reply on news.ycombinator.com