Comment by wokwokwok | Hacker News Reader

Comment by wokwokwok | Hacker News Reader

There’s so much truth in this.

It really cuts to the heart of it when you looking at the “devops cycle” diagram with “build, test, deploy” …and yeah, those other ones…

I remember being in a meeting where our engineering lead was explaining our “devops transformation strategy”.

From memory that diagram turned up in the slides with a circle on “deploy”; the operational goal was “deploy multiple times a day”.

It was about speed at any cost, not about engineering excellence.

Fired the ops team. Restructured QA. You build it you run it”. Every team has an on call roster now. Sec dev ml ops; you’re an expert at everything right?

The funny thing is you can take a mostly working stable system and make fast thoughtless chaotic changes to it for short term gains; so it superficially looks like it’s effective for a while.

…but, surrrrpriiiisssseeee a few months later and suddenly you can’t make any changes without breaking things, no one knows what’s going on.

I’m left with such mixed feelings; at the end of the day the tooling we got out of devops was really valuable.

…but it was certainly a frustrating and expensive way to get there.

We have new developers now who don’t know what devops is, but they know what containers are and expect to be able to deploy to production any time.

I guess that’s a good way for devops to quietly wind up and go away.

DevOps and releasing multiple times per day does not always mean to PROD and in most industries that is impossible. Continuous Integration and Continuous Deployment should and do mean to the development branch and container. Locally I can write code and skip all the tests, skip the linter, skip prettier, etc.….

When I do a commit prettier is run. When I PR into develop other dev’s review the code. When its approved and completed a build is run (including all tests and sent to Veracode and SonarQube for analysis). The build is deployed to the development container where the developers can smoke test and then when SQA is ready promote to Integration for real testing. Fail fast! If the tests fail, we can fix them. If code quality goes down, we can fail the gate === no deployment. Without DevOps and CI/CD these steps get skipped.

Not sure where the idea that software is released to production multiple times per ever came from. Yes, the develop branch should in theory be ready to be promoted at any time but we know that is not reality.

The big advantage to deploying to prod constantly is trunk-based development.

Any system where you maintain a separate development branch is one where you're invariably going to be asked to cherry pick feature B to production but not feature A. This is a problem because you tested everything on a version of the code where B follows A, but now B needs to stand on its own. Can it? Maybe. But you didn't test that.

With trunk-based development you have one main branch that everything goes to all the time. If you need something to not go out to real users then you put it behind a feature flag. If something needs to go out to users today, then you merge a change on top of your main branch that unconditionally shows the feature to users.

This way all code that ever makes it into production exists at a point along a single linear commit history, each merge commit of which was checked by CI and is the same history that devs developed against locally.

In my experience eliminating cherry picking makes a huge difference in reliability, and the only way to do that reliably is to constantly deploy to prod.

And yes, my current $day_job does this.

Congratulations on moving fast! If possible ‘trunk’ is best.

Long lived branches are a problem if you are doing hot fixes to prod. The goal is the process avoids hot fixes and critical bugs.

Some industries would not allow software to be released more than every few months. SQA needs to sign off. Users need to UAT. 2-week notice, 15-minute notice to log off. Downtime needs to be communicated to multiple time zones. Software versions need to be auditable for regulatory.

The only point I was trying to make is ‘publishing’ multiple times per day with a team of developers requires ‘devops’ and that release does not always mean to prod. The same setup for releasing to prod multiple times per day is the same as releasing to develop multiple times per day.

Long live DevOps

that doesn't work at scale though. the difference between dev, staging, and prod when there's a handful of services is fine. when there's 300 of them, and 200 of them are broken in the dev environment at any given time, that means you can't actually use the dev environment to do development in because the 300 other teams are also trying to do development in that same environment so their stuff is just as broken as your stuff.

so then you either have to accept that sometimes it's broken and just wait on it, or otherwise agree that staging should always have a working copy of Kafka, except that means the Kafka team can now no longer use staging to stage Kafka changes, so then they have to setup their own separate staging environment and then plumb sufficiently representative test data into that new staging, and then and then and then.

Development should never be broken, ever. Ready to ship and broken are two different things. By ready to ship it more like v2 has 8 total new endpoints and only 2 are ready, then 4, then 6, then 8.

When the tests fail, or the code quality goes down, the deployment fails. I would rather have DEV broken than PROD. Not sure how going directly to PROD would make anything better in this scenario. That is what Integration is for never broken always ready for promotion to prod. Plus a few broken dependencies means ‘yellow’ === impairment not unusable. If the app is unusable for any one dependency that is another issue.

Mocking data when you are ahead of another team is always the reality…

Just want to add that these are all good points and good discussion. Ultimately, I think we agree that DeOps is needed no matter how it’s used. The details of how exactly it’s used per product, per team, per organization is always ‘it depends’. Some of us need to jump through more hoops and there is no way any of our products would be updated in production multiple times per day much less multiple times per month unless it was a hot fix of a critical bug. CI/CD is something to strive for as if you can release to develop multiple times per day it’s only the surrounding process that prevents you from releasing to production multiple times per day.

> Development should never be broken, ever.

And devs should just not write bugs in their code, ever. Development mostly works, but is going to be broken in subtle ways that other teams are going to pull their hair out because their thing doesn't work, but their thing doesn't work because your team has broken development in a very subtle way that only that one team tickles. Your tests didn't catch it and the rest of your team didn't catch it during code review.

We have environments with thousands of services and it scales fine. Why would dev (or your lowest integration environment) be broken most of the time?

> except that means the Kafka team

That (a Kafka team, or a DB2 team) is a bit of a red flag for me. Many (but not all) “tech” teams like that are part of the problem. Cross functional delivery teams work much better, because running tech X in isolation is often not valuable.

The one exception to this is “x as a service” teams. Eg DBaaS teams, Or even a team that offers a pub/sub service. But in those cases such teams are literally measured by uptime SLOs, so if a pub/sub team can’t deliver uptime, they’d need to improve.

Where are you that has thousands or services and it scales fine? Twitter and Facebook famously both don't have a separate staging environment because they have thousands or services and it doesn't scale fine. They do canary releases and feature flags so as to do gradual deployment and testing in prod. If they can't solve the problem but you're somewhere that has, my next question is are you hiring?

Dev is broken because devs are doing dev on it. I mean, it generally works, but it's the bleeding edge of development so there's no real guarantee that someone didn't push something that doesn't work in a way that the rest of the company is relying on.

What is the DBaaS or pub/sub team's commitment to uptime in the staging environment? It's staging. if they have to commit to a reasonable uptime, they can't actually use it as staging for themselves. Saying they need to improve is trying to handwave out the fact that they need a staging environment where they get to run experimental DBaaS or pub/sub things.

I worked at AWS. Each team/service had their own alpha/beta/delta/gamma development environments, as well as one-box and blue/green production deployment environments, and deployed to waves of regions from smaller groups at the start to bigger groups at the end.

That all seemed to work reasonably well.

> Some industries would not allow software to be released more than every few months. SQA needs to sign off. Users need to UAT. 2-week notice, 15-minute notice to log off. Downtime needs to be communicated to multiple time zones. Software versions need to be auditable for regulatory.

As a person who works in a heavily regulated field ($$$), all the audit / traceability requirements apply. We still move fast (multiple releases a day) because we have demonstrated that it reduces risk and promotes stability — something regulators are very interested in. In concrete terms, faster release rates are correlated with lower sevs (or incidents). But the underlying cause is really that faster release rates cause teams to adopt more reliable delivery practices, such as automated testing, gradual deployments, and so on.

Eight years ago this was a hard sell to some regulators, no more. You still need to walk them through the outcomes but even in black letter jurisdictions it’s being accepted.

> Not sure where the idea that software is released to production multiple times per ever came from.

UX-focused development. Your concern is mainly about user journeys, acquisition tunnels, etc. So you rely on user feedback a lot. So you ship, A/B test, iterate.

Also, if your service consists of lots of microservices, you don’t want to wait to ship them all at the same time.

The most fitting example to both is probably Netflix. I guess they ship to prod every day a lot.

Fair enough.

Guess that is why Netfix is so Glitchy every other day j/k.

I should have kept that last sentence out but I will not edit it now as I asked….

It’s possible in many even heavily regulated domains, because there’s enough data now for regulators that it reduces risk.

Releasing multiple times a day in itself isn’t the point though.

The point is — adopting good practices to ensure trunk based development is pain-free. It’s about ensuring the release process is as automated as possible (if you’re releasing once a month, you are exercising your deployment process 12 times a year. When you’re exercising your deployment process 12 times a day, it really encourages you to work the kinks out.)

It’s also ensuring that your monitoring and alerting can keep up. Essentially it’s a signal to the entire org that “We will move fast. Deal with it.”

There’s data out there from even large non-FAANG orgs now about how this approach reduces sevs instead of increasing them.

I bumped into a DevOps job at Teradata, did it for two years, then left for where I belong, in gaming as a backend developer. It was a corporate with good pays and perks including international travel etc. Some friends are still there doing DevOps very happily. Talking to them it never feels a like a dying field.

I hated my DevOps role so pardon me for only reading the headings of the article, but talking about it in past sentences seemed very delusional.

The deep deep irony of hiring someone specifically to do "devops".

That's not devops.

Devops is when you have developers who are empowered.

When you hire someone specifically to do devops you are hiring an ops team and calling it devops.

So, yeah. That still happens, sure. However, I do think it's changing these days:

linkedin > connections -> control-F -> 'devops' => 0 hits

"Site Reliability Engineer" -> 10 hits

I think hiring SRE makes sense. I think hiring 'platform engineers' makes sense. People still do that, and will, as far as I know, far into to the future.

...but I think hiring devops means you never understood, even vaguely, what devops was; you're just using a buzz word in your job ad.

This. I see ads for DevOps roles as a signal that “this org doesn’t get it”.

DevOps is about people and culture. Your teams embrace automation, fail fast, lean — all the good stuff. DevOps is about concrete business outcomes. Not Ansible or bazel.

> hiring devops means you never understood, even vaguely, what devops was; you're just using a buzz word in your job ad.

Yes!

I honestly don't get this thread and the complaining. Yes, I was one of those empowered engineers at an org that did ALL of the DevOps. But do ya'll seriously think you don't sometimes need people who do ONLY DevOps? Cause I can tell you there are absolutely orgs that deploy dozens, if not hundreds of servers sometimes by region and in those cases you absolutely do need full time DevOps engineers to tune, configure, and deploy these fleets. And yea it's a full time job just to do that.

> in those cases you absolutely do need full time DevOps engineers to tune, configure, and deploy these fleets

At that point, it sounds like you're just doing ops work.

Correct, except generally they aren’t even good at ops. They tend to know a highly specific slice, or how to operate an abstraction. But troubleshooting Linux? Good luck.

> Devops is when you have developers who are empowered.

That’s just half true. If something doesn’t work in production, guess who can get mysql production logs? Not the developers. Can you access production dbs (not that that’s the way to fix things, but I have seen that in many orgs)? Nop. And if there’s anything you can do yourself (like changing things via TF files), you’ll need to wait until the gatekeepers approve your request.

The gatekeeper are usually called platform engineers.

But in any case, I don’t want all that power (and responsibility) either if it does not come with an increased compensation.

What parent commenter says that when you have "devops", there are no such gatekeepers.

I always got higher compensation in such companies/teams, because they weren't dying. At least, when I joined :) They needed efficiency instead of control and keeping salaries down.

Nobody agreed on what "DevOps" means.

A regurgitation of Sysadmins that knew some scripting?

Cloud Native Sysadmins?

Patrick Dubois said "a systems administrator working with an Agile mindset", and this guy started DevOps Days; which is where we get the term.

The "merged teams" spiel is from the "10+ Deploys a day" talk from Flickr, and it's but one definition of what devops means.

This is why I personally consider it failed, it's meaningless because it means different things to different people. You can never "do it wrong" if you don't even have a definition that everyone agrees on.

Reply on news.ycombinator.com