Hacker News Reader

Comment by namanaggarwal | Hacker News Reader

What I have seen in the companies that I have worked in

- Initially every engineer does everything

- Growth gives birth to independent SRE teams. Now we have engineers who does only SRE

- We realised SRE is bottle neck for product teams. SRE teams are removed and all SRE are embedded into the product teams to do dedicated work for them.

- Other team members are now required to learn SRE stuff, especially now that terraform etc are there so we are back to everyone is a SRE

note that FAANG level companies go back to independent infra, SRE etc maybe because of niche and custom softwares they have.

View on HN

Is it, uhh, common knowledge that the important thing isn't how the team is organized or how they exactly do their job but that there are engineers paid and incentivized and empowered to keep the status quo running flawlessly (or as close there to as possible) rather than being paid and incentivized and empowered to launch cool new features or make the money graphs go up?

The problem at most companies is that if you have two engineers, Alice and Betsy, and Alice launches a change that generated marketing buzz that gets spun into sales, and Betsy launches a change that prevented the site from going down over the seasonal rush, Alice gets promoted and Betsy gets told to work on more impactful projects.

Which isn't to say that Alice's work isn't important or promotion worthy, but you need an incentive structure that rewards both.

In my experience, Alice moves on to a cool new project to greenfield while Betsy gets stuck supporting Alices creation until they eventually get on-call burnout and quit. Alice checks in only when there's enough noise made either by angry customers, cto or angry SREs.

That's my 10yrs+ ops experience, at least, and this is hugely exacerbated if Alice is put on such a high pedestal that they don't even have to be on-call with Betty at any point of the lifecycle.

I think ultimately this dynamic is why Google can't keep most products around for longer than 5 years.

That is not necessarily an unhealthy cycle...

Organizations succeed by delivering customer value. Re-orgs of teams to remove pain points and blockers is a sign of an organization flexible enough to handle their growth. Even if you go back and forth a few times between similar states, the larger point is that you are willing to do what is needed to continue to effectively deliver value to your customers.

>the larger point is that you are willing to do what is needed

They seem to be unable to organize and delegate work effectively even when it's needed and instead rely on polyglot-engineers to bridge that gap by overloading their responsibilities.

Going back and forth isn't progress or keeping up with growth.

I have a very similar view. But I see more and more companies trying to build a Google-like SRE structure, where the team actually operates more mature services highly critical to the business. Tricky not to become traditional ops teams, but seems doable.

SRE becoming bottleneck with scale is something I see a lot. Usually regular product engineers that are more familiar with infra start doing it full-time for all the company.

It's great to know that we still have no idea what we're all doing in tech.

This matches my experience at my work. I am an early employee and we did one of everything back then, was told we had dedicated Ops/SRE and lost permissions to do stuff about half way through my tenure, and now this year it’s back to everyone is doing their own Ops/SRE.

That’s been over the course of about a decade.

cough back in the 80's I was employed by a division of British Telecom explicitly partially as DEV/OP on a billing system as I had SYSAD experience on the platform.

Later on mid 90's the BT World wide intra net was run by a full service team that handled hosting admin and development

> We realised SRE is bottle neck for product teams.

This means that management decided velocity is more important than reliability?

> > We realised SRE is bottle neck for product teams.

> This means that management decided velocity is more important than reliability?

No, given the organizational response ("SRE teams are removed and all SRE are embedded into the product teams to do dedicated work for them.") It seems to be like the motivation for Agile cross-functional teams instead of separate Analysis, Design, Coding, Testing, ... teams, and like DevOps, and like DevSecOps, etc., yet another instance of realizing that throwing products over internal walls in organizational handoffs produces inefficiencies.

I suspect the last one is the size of workforce, and even then "embedding in product teams" seems a common mantra mentioned from FAANG companies.

Reply on news.ycombinator.com