Evidence-based Software Engineering: based on the publicly available data
knosof.co.uk
knosof.co.uk
Evidence-based software engineering: book released - https://news.ycombinator.com/item?id=25073218 - Nov 2020 (118 comments)
Evidence-based Software Engineering – book beta - https://news.ycombinator.com/item?id=23769930 - July 2020 (22 comments)
Evidence-based Software Engineering [pdf] - https://news.ycombinator.com/item?id=3213482 - Nov 2011 (9 comments)
Evidence based software development - https://news.ycombinator.com/item?id=2707618 - June 2011 (1 comment)
What we actually know about software development, and why we believe it’s true - https://news.ycombinator.com/item?id=901600 - Oct 2009 (54 comments)
I'm pretty sure there were others in years past... maybe someone will find one.
We’ve taken a sort of controversial approach to software development in the public sector of Denmark because we don’t have enough resources. Small projects are build to run their own little lives with as little post-deployment development as possible. Because of this they aren’t build with best practices, whatever theories are the right way to do things in a current age, none of them are automatically tested because their functionally is so small it’s always going to be obvious where the flaws are and the only way for them to get additional development post deployment is if they turn out to be really good, and even then, we don’t expect them to live after 5-10 years.
It’s way of development goes against every theory on software development you’ll find being taught at universities and academies, I know, because I’m an external CS examiner at these places, and we’ve really only chosen to do this because we have 3 developers in an organisation with 10.000 employees.
Now the data shows us, that we’re actually better off doing the wrong thing.
That, sadly, won't scale.
> we're actually better off doing the wrong thing.
You aren't actually doing the wrong thing. At 3 employees the maximum complexity your team can handle is pretty low. It's also trivial to pass on institutional knowledge since well... the whole team can share one extra large pizza.
We never really need it to. That’s one of the few benefits of writing software in the public sector. We don’t need to expect an explosive growth in users.
We’ve seen 10.000 additional citizens move to our area over 5 years, so there is growth, but it’s far from everyone of them who would interact with a single solution that we wouldn’t buy from the private sector through public procurement.
There's a point where a team of three maintaining tribal knowledge is not enough to deliver certain project above a threshold of complexity. But you mention procurement, so I assume your team does a buy vs build pretty much systematically.
> That, sadly, won't scale.
You're contradicting the UNIX philosophy: make programs so small that they're inherently simple (or so simple that they're necesserily small). I think that's what the other commentor meant and that does scale.
I recommend skimming rather than straight-up reading.
There are chapters on Probability, Statistics, and Regression. None of those chapters illuminates the subject at hand; they provide an explanation of the author's techniques. It's background. There are several other "background" chapters, that aren't about statistics, but also aren't about the subject at hand.
So it looks as if I read most of the meat yesterday.
I read most of the chapter on Regression, because I didn't know much about it, and the author relies rather heavily on regression analysis.
• introduces a subproblem, that would be solvable but for one tool you're currently missing;
• introduces the tool you'll need to solve the subproblem;
• explains the tool in the process of using the tool to solve the subproblem. Only explaining as much of the tool as is necessary to solve the subproblem — more can be explained in later subproblems.
Sadly, I find that most books like this first introduce all the tools you'll need; teach them all thoroughly (including teaching stuff you won't need at all, because it's part of the explanation as traditionally taught); and then go about solving all of the subproblems using all of the tools, in a pretty random order. So the first half ends up boring, and the second half ends up messy and hard-to-follow.
It is from 2010, I don't know if their is really new things on the subject
A bunch of citations isn’t the same as evidence. When I dig into software studies, I find all kinds of issues, ranging from subjects of convenience (the professor used their undergrad class for their study) to industry studies that don’t define their terms. (e.g., MS did a large empirical study of TDD, but never defined how TDD was practiced—this is relevant given the misunderstandings of TDD that exist, particularly at Microsoft).
I didn’t read the rest of the book—just looked at the blurb, winced at the inclusion of McConnell repeating his 10x claim, and read his chapter—but my default stance is to be highly skeptical of academic software engineering, and even more skeptical of popularizations.
Edit: I’ve just glanced through OP’s book, and it looks like an impressive piece of scholarship. I remain skeptical of academia’s ability to teach much of interest regarding software engineering, though. (They spend a lot of words on Waterfall and traditional project management. Their words on the Agile-kinda stuff companies actually use today? One paragraph, paraphrased: “there is no data, we have nothing to say.” Not the authors’ fault, but sums up the irrelevance of academia to real-world SE, in my mind.)
But I'll happily go down that road. The 10x claim ("the most productive engineers are 10x more productive than the least productive") is ridiculous on its face; the least productive are zero or negatively productive. There are people who can't program at all, and there are people who actively detrimental. Therefore, two things are clear:
1. The correct statement is "the most productive engineers are infinitely more productive than the least productive."
2. This is useless information.
A more useful statement would be "the most productive engineers are x% more productive than the median." Or even a histogram—that would be super useful!
But, it turns out, there is no way to measure real-world engineer productivity, so any quantitative claims are, in fact, utter bullshit.
And, if you look at the original research that McConnell cites, that turns out to be true. IIRC, the study was comparing the how long it took to debug programs when using a time-share system vs. a batch-processing system.
(To be clear: I'm arguing about the claim, not the variations in productivity. Obviously, people have variations in productivity. But there is no rigorous evidence for the magnitude or nature of those variations.)
https://twitter.com/gvwilson/status/1326190553738977281?s=20
I appreciate the author's frankness about the limitations he's working under: if there's no data available about some topic, he doesn't discuss that topic. In fact he's pretty rigourous about 'following the data'.
I think I would like just a little bit more speculation about e.g. developer motivations. And it's a shame there's not enough data to support an analysis of modern development paradigms such as TDD, pair-programming and so on. Of course that's not the author's fault; anyway, in a sense, the absence of data is itself interesting.
I've found the book interesting. I think there are too many 'data dumps', i.e. descriptions of datasets that don't cast much light on anything.
I think I'd find a review or abstract that distils the book useful and engaging.
Can you names the others?
I see a ticking time-bomb and the blind leading the blind. Not sure if those are canonical.
About three dozen other vignettes that I definitely can't pin to anti-patterns. Like man-eating plants, teaching magic to rabbits, gearheads, touching a lightbulb... someone please help.
Amusing and revealing at the same time. Bet they did not take long to agree on that senrence
Hahaha, classic. Like his style in general.