Comment by roenxi | Hacker News Reader

Comment by roenxi | Hacker News Reader

A few related observations:

Software development is not a stable process. Either a team is always building new things in which case there isn't a consistent process to measure, or there is a see-saw as people release new features, deal with the bugs in the features, then go back to building - that isn't a controlled process, it is going to oscillate in statistically weird ways.

If SPC is applied to bugs, it will be monitoring the relevant manager's habits. That is to say, if you show me a nice in-control timeseries of bug resolution, all that says is when a bug blows out horribly the manager splits it into 2x tickets or something similar. It isn't necessarily a bad outcome (small tickets are happy tickets and gently stressing managers is a good idea) - but don't expect the devs to behave differently.

It is good to have a grounding in SPC, just don't try to apply it to every timeseries that you see. Bugs are a timeseries, but they aren't expected to be a controlled process so SPC's assumptions break down and the logic doesn't work. If it does work, it is probably measuring something other than the software development aspect of the process.

Things in software development that have been stable in my experience:

- Weekly deploy count

- Weekly growth in lines of code

- Fraction of candidates hired

- Weekly number of bug fixes

- Weekly number of problems with a third party collaborator

- Many internal metrics generated by the software, reflecting usage etc.

- Weekly number of consultant hours required

- Monthly growth of feature flag count

- Length of standup

- Time required to complete "small" tasks (i.e. those that don't involve novelty)

- Length of successful build in CI

- Proportion of CI builds that fail at least one test

- Growth of number of tasks in backlog

I could go on. The point is that while much of the value of product development comes from novelty and variation, there are many parts of the process that remain the same from week to week.

Figuring out what these parts are and getting variation out of them allows the developer to

(a) focus creativity on systematically solving process problems instead of doing it ad hoc, and

(b) let the process recede into the background and focus creativity on creating end-user value.

Your heartbeat and breaths/week are also quite consistent. The issue here is that you have to have a reasonable theory of why variance in the metrics you're tracking will destroy value. And that means actual value, not whinging that standup goes too long. I like a short standup as much as anyone, but if that is a material driver of value destruction then your organisation is not ready for statistical quality control. Plus it probably goes over time consistently.

And once you start looking at the value, the lesson of software is that high variance activity is often the value add. It is the week(s) where someone implements pivot tables in Excel that creates probably billions of dollars in value over all of humanity. If that turns up as a statistical anomaly in the metrics because their line manager didn't bug them to fix bugs that week, that is a problem with the metrics not the programmer.

This isn't a Toyota production line (if you're interested in the history of this, that is no random example) where value is uniformly created with each car and optimising the daily process down to degree n creates value. This is software. The value isn't created in the same way and these tools are not powerful in driving value add decisions. Variance is untidy but by no means an enemy. It must be managed case by case in context.

> The issue here is that you have to have a reasonable theory of why variance in the metrics you're tracking will destroy value.

Establishing this theory requires a stable process! Without a stable process, you cannot make deliberate, systematic changes and observe how it affects outcomes. That sort of observation is key to theory-building.

I agree with most of what you say about the value in product development coming from innovation which is literally unaccounted-for variance. I just don't think that innovation happens in the length of standup meetings, which I think is better controlled statistically.

> Establishing this theory requires a stable process! Without a stable process, you cannot make deliberate, systematic changes and observe how it affects outcomes.

shrug Welcome to software engineering. Enjoy your stay.

If anyone has figured out how to make deliberate systemic changes that add value, they really need to publicise it because I'm not aware of many approaches that aren't absurdly basic (things like get release and get fast feedback).

There are lots of examples of companies like Google, Microsoft, AWS, etc that generate huge amounts of value from early on in their life cycle them then coast on that while experiments with processes never really move the needle all that much. Google has been search and ads for 20 years and none of their experiments with software quality since then have been all that impressive. It isn't even all that clear from a customer perspective that the quality and quantity of software is improving. If anything, they need to slow the programmers down, write less code and dedicate more resources to maintaining platforms and pushing them to succeed. SPC won't help with that though, because measuring "nothing happening" isn't what SPC is targeted at.

In fact, open source projects where there is usually no QAQC in sight are vast wealth and productivity fountains, we have people like Hipp over at SQLite who just tests everything to within an inch of its life. Good luck replicating that level of productivity with SPC. No systemic process has come close to matching the productivity one madman with a love of databases and stability. We have no idea how to make that repeatable; because the fundamental process that creates value isn't stable.

> I'm not aware of many approaches that aren't absurdly basic (things like get release and get fast feedback).

A bit much to call that “absurdly basic” when release cadence still varies massively even among successful companies. Patch Tuesday? Why not Patch Every Day!?

There's a difference between input and output metrics to be considered too. Attempting to manage the output metrics directly rather than addressing the underlying causes is almost always the wrong thing to do.

> Software development is not a stable process.

This causes confusion in my experience. "Stable" doesn't mean what it does in casual usage. It means that you have a mean and variance that are not varying over time (and there are ways to alert if they are). A stable process can lurch around like a drunken sailor and still be considered stable for SPC purposes.

I think bugs should be stable if you normalize it by a size metric. So bugs/SLOC or bugs/story points.

They aren't stable. Bugs, viewed as a timeseries, don't have a steady mean and standard deviation. There is a feedback loop where those measures change depending on the maintenance and development strategy of the software, and priorities are going to shift suddenly. If there is a fix-small-bugs month for example, or a fix-this-client's-bugs push, or a fix-the-bugs-in-the-new-feature campaign. The statistical properties of all these are not identical.

And once you've normalised for story points, what you're measuring is stability of the story point allocations. Which is to say, different managers will get different means and standard deviations even if the devs change nothing about their work habits. It isn't measuring the devs, software quality or even real number of bugs; it is measuring the ticket creation process.

As long as you know it isn't measuring the software, all is fine. But people do get confused.

> If there is a fix-small-bugs month for example, or a fix-this-client's-bugs push, or a fix-the-bugs-in-the-new-feature campaign. The statistical properties of all these are not identical.

For sure -- but these are surely special or assignable causes of variation. That's one of the key insights in SPC: that some variations are ordinary and some out of the ordinary. A "fix the bugs" campaign is out of the ordinary.

> Which is to say, different managers will get different means and standard deviations even if the devs change nothing about their work habits.

If they have a bad measurement rubric, certainly. But that in turn goes to questions of gauge reliability, a topic about which SPC has a lot to say.

I should say by way of agreement that I'm not convinced that SPC can be safely and effectively applied to software development processes. But I'm also not convinced it can't. I don't have a firm, final position on the matter. I would need to apply it for a while and see for myself.

> For sure -- but these are surely special or assignable causes of variation.

Well, for starters they are process changes. The team is working in a different way to normal and, as expected, getting a different result. So it is special variance, but because management has decided that the team should adopt different processes with different statistical characteristics. These aren't one-offs, this is routine for how a well managed software team should perform. Priorities ought to change as the situation develops. And regardless, regular variations are what indicates that a process that isn't under control. If a process is under control, variances should be remarkably rare.

Every month in software engineering is out of the ordinary. That is why SPC doesn't work; there isn't a repeatable process to model. There are lots of different processes chained together as vague specifications are continuously translated into logically formal specifications in an environment of continuous renegotiation. It might be possible to make a software team work in a controlled way, but it is pretty stupid - either the team won't fix some bugs to make the metrics look good, or they have to fake bugs from time to time to make the metrics look reasonable. Both are inferior to non-statistical controls of prioritising bugs with reference to how challenging the fix seems likely to be.

Say we're going to implement an application. The bugs aren't going to appear as a stable process, there'll be some sort of big wave up front, then successive waves as new features are identified. Eventually the application will be more or less finished and the engineering team will clean up the long tail of bugs. The mean and standard deviation of the work aren't predictable in advance and aren't in control as far as SPC is concerned, because those tools assume steady state. Smoothing that curve from a hill-shape to a flat line is not value-additive; it is bad management in its own right. Resources should be reallocated to fix bugs after a big release and then moved back to dev work.

Now there are going to be some situations where SPC isn't crazy, but it is still a bad management tool for software because the SPC teams aren't going to be agile in the face of change, because all their tools will scream blue murder at them for no reason.

I have enjoyed a different experience of software development: the team breaks down stories into units of roughly equal complexity. Bugs get fixed first and do not contribute to velocity.

Over a medium time frame I've seen this approach produce remarkable stability in velocity measurement.

Note that this works by allowing product priorities to fluctuate, but not allowing quality to be a dial that can be turned. A distressingly rare configuration of power.

> "Stable" [...] means that you have a mean and variance

And software development, as a process, does not have a (finite) mean or variance.

> I think bugs should be stable if you normalize it by a size metric.

Bugs approximately follow a heavy-tailed distribution, such that as the number of samples increases, the empirical average work to fix (and possibly other measures of severity) does not converge to any finite value, but instead increases without bound. (I think roughly logarithmicly, but don't quote me on that.)

In particular, it doesn't satisfy the requirements of the central limit theorem, so a lot of statistical techniques work poorly or not at all on software projects that are large enough, individually, to do statistics on (as opposed to doing statistics on populations of software projects, which seems to mostly work, usually).

> Bugs approximately follow a heavy-tailed distribution

What's your data on this? In my experience, bug fixing time has been surprisingly thin-tailed. (High variation, yes, but not subexponential.)

My experience here is also mirrored by Avery Pennarun, who observes that "every bug is the same size": https://apenwarr.ca/log/20171213

> And software development, as a process, does not have a (finite) mean or variance.

Ever heared of something called linear regression? E.g. LOCs/week might be an interesting enough metric for a growing company.

Edit: better example

Reply on news.ycombinator.com