Hacker News Reader

Comment by georgeaf99 | Hacker News Reader

Hey y'all, I recently started working on a CI/CD product at an organization where the number of deployments per month is in the same ballpark as Netflix. In my experience, adopting systems for rapid deployments also requires adopting new processes. For example, products that are deployed frequently need to have developers on call during deployment windows. These developers tend to have a specialized skill set that makes them effective at handling botched deployments and bugs in production.

How y'all are planning to approach the cultural changes companies will have to adopt in order to leverage faster deployments?

View on HN

We believe that DevOps encompasses 3 areas that must change: people, processes, & technology. We’re seeing evidence that the people/culture is already changing organically because companies must change in order to compete in the software era, and the velocity at which we see software-first companies entering traditional spaces like banking/finance, real-estate, hotel is also accelerating.

That said, in order to make this rate of change to happen faster, instead of trying to convince people through debates, we have to show proof that moving towards a true DevOps model is successful, not just because you get to deploy more, but because it matters to the business. We want to show that CI/CD helps meet SLAs, new valuable features are added and allows the business to address new markets.

To add on to that: The best way we've found to work with large enterprises (aka Global 2000) is to start w/ just one team & app, show success with that, and then expand from there.

We've found that big companies need to see one team achieving success before they'll adopt it throughout the organization.

We've employed this approach successfully with our first several customers. They're going from from one app --> 5 apps --> "GA" rollout internally.

I just founded my third startup Elastic Byte (https://elasticbyte.net), which is a DevOps and infrastructure consulting company operating as SaaS (public monthly pricing plans).

Indeed culture is a big part of the DevOps puzzle besides software. This mentality shift is also a part of what we offer (training and advice). Individual ownership, moving fast, are all parts of the greater strategy. I've found the talk "You shipped it, you fix it" [1] by Ron Cohen (CTO of Opbeat) a great source of solid ideas.

https://opbeat.com/community/posts/you-shipped-it-you-fix-it...

The old model where developers hand off their code to the ops department is fundamentally a bad strategy because it leads to:

   "It worked fine in test... Ops problem now!"

There is no accountability. If your code causes problems when deployed to production, ops is the one getting a call/page in the middle of the night, and trying to fix the problem for which they have no domain experience or understanding. However, if you deploy your own code at your own pace, then there is accountability.

Yeah that mindset shift was the hardest part of transitioning from a traditional ops to a no-ops approach at our last company. The technology has no chance if the culture isn't ready.

We especially see this with F500 companies. They insist on multiple manager approvals for deployments. CI/CD is really scary to them because it feels too risky.

One of our prospective customers (at a F500 company) told us about how they have a cross-functional deployment 'swat team' that all gather in a room on a Friday night and literally press a button to deploy the update, then see what happens. They take their website offline in anticipation of errors (and put up the '90s construction worker logo) and then their ops people get paged over the weekend to deal w/ errors. Just ridiculous!

The irony is that the slower deployment velocity creates more risk, making deployments scary affairs. It's a terrible cycle. Deploying 7x per year means each deployment contains many more features/fixes, which introduces more risk. Vs. breaking a monolith up into microservices, and letting each component deploy at whatever cadence is best for it. And with Spinnaker each deployment can be traced back to the original git commit hash / jira ticket / etc., creating much more transparency in the deployment process.

We’ve seen a few fundamental changes in culture across the board.

1. Product engineers are embracing ownership, and despite being on-call, they are excited at the prospect of being able to deploy their apps whenever they want.

2. Ops teams are transforming into engineering teams. When you start to automate everything (monitoring, deployments, testing, etc), it frees up ops folks to work on more interesting problems within an organization.

Reply on news.ycombinator.com