Dev productivity metrics suck. Ops reviews are key for AI-accelerated eng orgs | Hacker News Reader

Dev productivity metrics suck. Ops reviews are key for AI-accelerated eng orgs

cortex.io

11 pointsgsdatta10 comments

I'm trying to disentangle this from the established / proven / trusted "dx core 4" (ask your local devops person if you don't recognise the name).

I found "initiatives" was added. What does this new initiatives measure bring. Why do we care about otherwise unqualified initiatives, how do i know that doesn't just mean using the other 4 proven measures as cover for pushing pet projects without merit?

I'm super cynical tonight it seems. This is just rubbing me up the wrong way i guess and i can't really put my finger on why.

I know I'm biased but Core 4 (and similar) rub me the wrong way – measuring individual developers as the atomic unit IMO is always meaningless. It's a proxy for organizational health but not directly correlated.

Especially now with AI, what do metrics like "prs/engineer" even mean when you have background agents open/reviewing/releasing PRs without human intervention? what is the right unit for measuring health of the org?

FWIW I write a lot more about "why not existing frameworks" in depth in the full paper.

Initiatives are defined specifically as non-productive, technical leverage-producing initiatives that affect the org's health as a whole and are often left behind. For example, we recently ran an initiative around feature flag cleanups and full rollouts that we tracked religiously during our weekly OpEx review – without which we probably would not have had the same success with that cleanup initiative.

I understand your cynicism with "yet another framework" but (and I know I'm biased) this framework is intimately tied to ops reviews as a mechanism for both measurement and organizational change.

> Core 4 (and similar) rub me the wrong way – measuring individual developers as the atomic unit IMO is always meaningless

And From: https://getdx.com/research/measuring-developer-productivity-...

    > Diffs per engineer*
    >
    > * Not at individual level

Are you guys agreeing or disagreeing with each other?

Candidly, we're disagreeing.

The framework is explicitly titled "Measuring developer productivity with the DX Core 4". Diffs/engineer, even if not at the individual level, is still meant as a proxy for individual throughput. The question they are trying to answer is "how do we make Ganesh X% more productive?"

My argument is that that is the wrong question to be trying to optimize for, regardless of how you measure it. It also does not capture the risk dimensions with AI adoption (and the reduction in humans-in-the-loop) which my paper tries to directly address.

Dev productivity metrics frameworks are (and always have been, IMO) bad for measuring organizational effectiveness. Operational Excellence reviews are the future of measuring and improving organizational health, and I wrote a paper making the case for this + a framework that helps make the OE review effective.

That sound interesting. Would you mind sharing it?

Should be available for download at the link in the post!

https://www.youtube.com/watch?v=6_XSShVAnkY

I'm the author – I spent 6 months writing this myself, and, if it helps, our product doesn't even do the stuff in the framework. I'm not sure what I'd be selling?

Their web site has a missing image.

Reply on news.ycombinator.com