Evidence-based software engineering: book released

However, the other groups were much less factory-ish, though rarely anything truly novel. Only a small cadre of programmers were working on anything that really required novelty and creativity.

It's a spectrum and that has to be understood by all participants in the discussion. Managers want everything to be like that first group, because it's so consistent and predictable. They want to know that a project will take 1000 man-hours and be right 95% of the time. Many programmers want to see themselves part of the last group. The reality is, most of us are in the middle. There are aspects of the job that are almost mechanical, and aspects which require greater creativity or "craft". If we can clarify that for the managers, it can go a long way to ending some of the nonsense.

This is a big mischaracterization of what "software engineering" is attempting. In this case it is an attempt to document what should be learned from experience and intuition. Because, the vast majority of software engineering IS "factory" work because its the grunt work of building what should be a well understood system, with well understood tooling. This doesn't mean that there isn't plenty of space for creative problem solving, particularly when the system is underspeced, it just means that the correct solution most of the time is the boring one.

Most companies don't want "software artists" anymore than they want "artistic bricklayers" or "artistic aerospace engineers". What they want is predictable, maintainable error free software that still works when the "artist" moves on.

That doesn't mean that your not free to upload a ton of artistic software to your github, or have opinions about how something should be designed. It just means that a professional should choose the accepted method over the fun one when given the chance. And why shouldn't they? In very few cases are "software engineers" being paid to work on their own pet projects, the end result is going to be something that the company owns and is responsible for, not the individual working for said company.

I've seen a lot of the opposite. Yes, coding is a design practice, but I've had to clean up a lot of messes resulting from just plain bad design because nobody involved - generally ~25 year olds with very little experience out of school - knew that there were lessons from the past they could learn about what designs would and wouldn't work.

I agree with you that programming is an endeavor that benefits from experience, and wish that people would realize that means they can learn from the experience of others. Sure, intuition is involved too, but one common thing I've seen in shitty code I've had to salvage is that people often don't apply their intuition to "how could this code fail" or "how easy will this be to modify in the future"?

That said... taking a look at this book... I don't see much in the description or table of contents that would teach those folks whose work I'm decrying above much useful about writing good software. It has sections on reliability, project estimation, and development methodology as separate things - plus a lot of non-software-design stuff. But to me, the flow is different - estimation, reliability, and delivery will all suffer if you don't have the right fundamental design skills. You can't get much better at any of those without some deeper underlying changes.

It seems to have a lot of discussion of studies adjacent to software-related things, but I'm not sold on them saying much meaningful about software design.

There are a few high-level take-aways from the recurring patterns seen in the analysis; these include: • there is little or no evidence for many existing theories of software engineering, • most software has a relatively short lifetime, e.g., source code is deleted, packages are withdrawn or replaced, and software systems cease to be supported. A cost/benefit analysis of an investment intended to reduce future development costs needs to include the possibility that there is no future development; see fig 4.24, fig 5.7, fig 5.52, fig 5.69, fig 6.9, fig 3.31. Changes within the ecosystems in which software is built also has an impact on the viability of existing code; see fig 4.13, fig 4.22, fig 4.59, fig 4.61, fig 11.76, • software developers should not be expected to behave according to this or that mathematical ideal. People come bundled with the collection of cognitive abilities and predilections that enabled their ancestors, nearly all of whom lived in stone-age communities, to reproduce; see chapter 2.

Ex: Using language Y for X was a disaster because of Y didn't have hardware acceleration and we were unable to reach goal Z. Before attempting to use language Y in production again, make sure platform support has improved.

Speaking as someone who has observed the kind of massive technical debt you can incur from letting your programmers mature one mistake at a time and ten reinvented wheels later, I would certainly not be too quick to dismiss the proposal.

> "The labour of the cognitariate is the means of production of software systems"

I fail to understand why you would disagree here. That sentence to me means that producing software systems requires brain work. Whether thinking, designing, coding, testing, debugging, reviewing, it's all part of the "labour of the cognitariate". Compiling isn't.

Yet, everyone has fun playing chess and even Grandmasters continue to be mesmerized by it because of the complexity - there is always something new to learn. Also, GMs have access to the top chess engines and can be studying using the same engine but have radically different styles.

They live for the parts of their games that are novel and so do we.

If all of our development was rigidly defined, we would just automate it. We put up with the boring parts of our development because we know there'll be parts of it that we can design.

If you are a hobbyist developer or a student, by all means, explore on your own and learn from your own mistakes. But that is not efficient if you want to get to the forefront. No GM today will refuse to learn opening theory or study the endgames.

We will still create magic and have fun.

These days, with virtually unlimited memory, storage, and extremely high software reuse, the industry has been effectively bootstrapped to the point that the average software engineer is more of a tradesman than an artist - a plumber or an electrician, who's job it is to fit standard components into the space provided for by the business logic. While there are elements of creativity to that job, it's no longer a truly creative medium for most jobs. This in a broad sense was the miracle of Java.

That's not to say that there still isn't the room for beautiful, elegant, code. But those tasks are rare, and they tend to target some low-level highly reusable code that the digital plumbers can reuse over and over without any nod to artistry.

If assembly line levels of engineering are ever achievable in software engineering, then it's most likely that software engineering can simply be mostly or entirely automated and the "engineer" disappears. But that's not where the field is yet. When that's achieved (maybe in the near future!), a user of a computer can simply ask the computer to build software to perform some function based on some conversational Q&A and and entire, scalable software stack will appear out of the ether.

"Computer, I need to..." and software will appear that does that.

> Striping developers of their autonomy is the primary cause of poor performance, not an inability to execute so-called "best practices"

Citation needed for this one, I've worked with more than a couple mid-level and "senior" developers who I would have sworn had never read a single article about best practices for stuff they did every day.

> In software, production is negligible. Which promotes the misconception that developers are producing software. In actuality they are designing software. The difference may seem subtle, but it is crucial.

This I agree with entirely but only for probably the top 20-25% or so of software. (Top as in most creative or most novel, not best).

He mentions at the beginning of the probability chapter:

"Readers are assumed to have some basic notion of the concepts associated with probabilities, and to have encountered the idea of probability in the form of likelihood of an event occurring"

This doesn't give us any insight into what background is expected here -- do I need to have taken a measure theory based probability course? Or would a calculus-based course suffice? What about a course for non-STEM majors? The word "basic" means different things to different people, and is generally an anti-pattern in mathematical writing. It's meaningless at best, and gatekeeping at worst.

Then he launches into this downright strange rabbit hole about mental models. None of this discussion in this chapter feels properly motivated, the writing is disjointed, and it reads like an unedited, non-proofread stream of consciousness. In particular, the exposition gets too distracted by its own examples.

Don't get me wrong -- many of the paragraphs in isolation are interesting, but taken together it's excruciating to read. It's an impressive project overall, and certainly ambitious, but it falls down due to the writer's lack of empathy for the reader. I suspect the situation would be better if they reduced the scope of the project and kept the writing more focused.

That’s what I was looking for when I clicked the link here. Do you have any recommendations for that kind of stuff, evidence-based if possible?

It's quite difficult to talk about empirical software engineering without discussing methods, after all papers like [1] were deemed necessary 20 years ago and still the occasional meta-paper is published about correct design of experiments or analyses. As someone who worked in the field it doesn't seem particularly surprising to see some treatment - there are a handful of papers in my former subfield that are oft-cited because they describe a statistical procedure/experiment design consideration, but they also bundle the explanatory stats "for free".

I would hazard a guess that the intent of these chapters is to equip the reader with enough background that they could replicate or run some of the experiments in the book to try to specify findings/experiments to their own organisations. I'd follow that with an assumption that the author felt that chapter 13 needed background, and recursed until they'd finished writing a textbook.

[1] Kitchenham et al. "Preliminary guidelines for empirical research in software engineering" 2001: http://www.ehealthinformation.ca/wp-content/uploads/2014/07/...

I used to think that sensible and compelling new ideas would be adopted quickly. Today, I believe this view is hopelessly naive. After all, it took 40 years before we recognized the power of the Toyota Production System.

I read this book on the strength of some recommendation here on Hacker News. It is quite relevant to this general topic:

https://www.goodreads.com/book/show/6278270-the-principles-o...

    while(true) 
    {
        println("hi")
    }

In other words, we hardly know anything

There are 2000 footnotes referencing published studies.

It's not perfect, but hopefully there are enough different views all triangulating on "truth" to provide value.

The world is a complicated place and the truth is hard to find. Unintentional confirmation bias is a thing. Intentional (at some level) cherry-picking to support your preferred answer is a thing too. But that doesn't mean that nobody ever learns anything from looking at data.

If someone claims to be "evidence-based", then their intention should be given the benefit of the doubt. They may be affected by confirmation bias, but that doesn't mean their conclusions are wholly baseless. You should take a look and see how they're doing before dismissing them out of hand.

When the data presented to us suit our current view or agenda, we somehow feel that it's valid. I remember when discussions about "open plans" or "estimates" were still a thing, without fail "evidence based" works such as Peopleware or The Mythical Man Month would be mentioned.

If this book turned out to agree with the view that "scrum is not really more productive", or "less meetings means more productive", or "the interview process is broken", then we'd all be really glad to refer to its data.

By assuming there's a "right" way to arrive at conclusions, people run the very dangerous risk of believing they've got the One True Answer, and everyone else are Wrong.

I think reality is a lot closer to the idea that there is no objective/rational way to decide what information is relevant and what information isn't, and we have to constantly adjust that dial in everything we think about and do.

There is an article about a group of people who have greatly reduced emotional capacity (not sociopaths, something more severe), and are therefore more or less rendered incapable of making decisions. It's a fascinating read, but I can't find it in my brief searches. Maybe someone else knows what I'm talking about?

Edit: I found the relevant anecdote! From an NPR interview https://www.npr.org/templates/story/story.php?storyId=101334... about a book called "How We Decide":

"And I think one of the best examples of this comes from the work of a neurologist named Antonio Demasio, who in the early 1980s was studying patients who, because of a brain tumor, lost the ability to experience their emotions. So they didn't feel the everyday feelings of fear and pleasure. And you'd think, if you were Plato, that these people would be philosopher-kings, that they would be perfectly rational creatures, they'd make the best set of decisions possible. And instead, what you find is that they are like me in the cereal aisle, that they're pathologically indecisive, that they would spend all day trying to figure out where to eat lunch.

They'd spend five hours choosing between a blue pen or a black pen or a red pen, that all these everyday decisions we take for granted, they couldn't make. And that's because they were missing these subtle, visceral signals that were telling them to just choose the black pen or to eat the tuna fish sandwich or whatever. And then when we're cut off from these emotional signals, the most basic decisions become all but impossible."

Of course, I haven't worked everywhere and I recognize that this opinion is based purely on my own experience. But companies that generate billions upon billions in revenue seem to care less and less about functional process improvement and more about short-term profit gains. Because ultimately, the C suite is paid based on stock performance for most large companies, and that becomes the only metric that matters. In that situation, everything falls in line behind it.

If your engineers can't deliver value because of say product, who cares how productive they are? Measure business outcomes - you will get what you measure. If engineering is being measured by the ability to deliver business outcomes but engineers can't because of other people, the software engineers are going to let you know* and it's an important thing to learn about and fix.

*this assumes you have competent managers and engineers who are willing to communicate with those managers, but if you don't have that, this is sort of a moot point.

- Deployment frequency

- Lead time for changes

- Failure rate of deployments

- Time to restore service

These collectively capture some sense of "quality and quantity" of changes.

What they don't capture, obviously, are whether all changes are meaningful and valuable to the customer.

However doing well on these metrics is nearly a prerequisite for doing a good job of bringing the voice of the customer into the process, so maybe trying to capture customer value created is step 2.

Anecdata and perhaps unfair but from my own observations, once you get to C-level, who is doing the communicating is much more important than how the message is conveyed.

There is definitely a 'club' mentality, usually the way you will get a voice is by building relations with C-levels that will 'adopt' you as their protégé/mentee. Once you have someone in the room who can say "we should listen to this person" then you will have a chance of actually getting info/influence upstream.

But to be build those relations there is a fair degree of 'self-promoting' and being seen as the go-getter in the crowd. Silently doing your work and silently going above and beyond is usually just a recipe for disappointment. Another thing I don't often see discussed is that, as with a lot of other things, luck and timing matters a hell of a lot.

That's the 3rd sentence in the linked page.

It sounds like you're saying that PhDs should decide what best practices are, without regard for evidence. If so, that had better be sarcasm, because it's patently absurd.

If your point was something else, could you clarify?