Scaling and Developer Productivity (2016) | Hacker News Reader

Scaling and Developer Productivity (2016)

blog.coinbase.com

43 pointskaushikt16 comments

>>> As we started to deploy all of our services through Codeflow in July we peaked at 27% of all deployments failing.

This strikes me as a lot. I can't imagine one quarter of all deployments failing. Does it means cases where the application doesn't start in production, like due to a typo in the source code?

Especially at a company which is essentially acting as a bank, powering non-reversible crypto transactions.

I don't think, as a manager, I'd be prioritizing statements like "we encourage new engineers to deploy coinbase.com on their first day" if I worked at a financial institution.

"Move fast and break stuff" has no place in FinTech or financial services (or many other critical industries).

I wonder if the higher ups are doing risk assessments on their processes. I totally understand the benefits of CD done right, but a quarter of deployments failing must show up on come IT compliance / audit team's radar as a risk that needs mitigation (Especially for a financial institution!!!). Hopefully we're just missing some context.

Surely relying on deployments as a measure of developer productivity is pretty flawed. You could deploy the same code over two deployments instead of one or deploy code that has no meaningful impact on the product and the metrics would look good but actual value added would be low or zero.

Deploying small changes derisks large outages. The smaller the change, the easier it is to revert it. This in turn translates to less time spent debugging and panic-shifting priorities to fix production deployments. Therefore, there is a relationship. As to low value contributions, their consensus process seems to filter out the extreme cases.

This is where metrics and KPIs part ways. Metrics are often poor proxies for the thing you actually want to drive, but a thoughtful leader can easily mitigate that effect by monitoring several metrics and understanding how they combine to form a more holistic picture.

That can all fall apart rather quickly as soon as you turn something you're measuring into a performance indicator that will be imposed on people as a way of assessing how well they're doing their job. If there's any room at all to game the numbers, then you've just created a huge incentive to do so, and it will happen. At which point the thing you're measuring has stopped being a metric in any meaningful sense of the term.

I can think of a good example of this. I worked at a fast food chain in high school. Our goal was 30 seconds from the time a car pulled up and started to order over the loud speaker to the time they pulled away w/ their food. We would rush the customer through the ordering process to the detriment of customer service and often have them pull over to the side to collect their order to cheat the clock.

I used to work in the kitchen at a restaurant that did similar things with time to get food out to tables. I don't know exactly how much beef I wasted due to keeping anticipatory burgers on the grill, even when I had zero open orders in the queue. All I know is, when I die, I'll have to have a very uncomfortable conversation with St Peter while a whole lot of angry cattle are scowling over his shoulders at me. Such a waste.

If you're trying to build a system and a culture that deploys frequently and often, this is a great way to do it. You want engineers deploying smaller changes more frequently - smaller changes are easier to review, easier to reason about if something goes wrong and easier to revert if they need to be.

You want to incentivize people to deploy as often as possible because it forces people to address the pain points of deploying. The idea is you build a cultural idea and drive towards a goal, in this case deploying on the first day, and use the cultural goal to address the technical pain points.

All of this seems odd for coinbase though. We did stuff like that at Etsy, because the key insight was that it's cheaper to fix errors than it is to prevent them in an e-commerce context. A hiccup on an e-commerce page just forces a reload and it's usually not a big deal. A hiccup in a bank transaction is much more frightening.

The same code split across two pull requests is preferable to the rest of the team (assuming they are actually trying to understand the PRs). Sort of a clever way to incentivize that.

I'd argue its a way to _hopefully_ incentivize that. Number of deploys has no inherent value though, so optimizing it opens you up to pointless progress.

How about, instead, you optimize _time to deploy_. i.e. the delta between code submitted for review and code running in production. Then, who cares if we didn't deploy 5 times today. Maybe we didn't have 5 things that needed updated.

For a second there I thought it said Scala and developer productivity.

whats with the sudden coinbase articles on first page?

Good article from blog gets published, people keep reading other articles in the blog and more stuff makes it to the frontpage. Happens from time to time.

It should be noted this article is from 2016.

Reply on news.ycombinator.com