Ask HN: Best empirical papers on software development? | Hacker News Reader

Ask HN: Best empirical papers on software development?

130 pointsKingOfCoders31 comments

There are some good empirical papers, but I only know very few. What is your best empirical paper on software development?

View on HN

These days I've been really keen on qualitative studies, where scientists work on building a better understanding of software instead of trying to validate theories. Some examples of that:

https://link.springer.com/article/10.1007/s10664-016-9464-2 — scientists followed a team of developers for three years and recorded all of their sprint retrospectives and people kept forgetting what they already learned.

https://jlubin.net/assets/oopsla21.pdf — researchers compiled 23 hours of Zoom sessions and 15 hours of Twitch programming livestreams to study how people write code in static FP languages.

In addition to that, there's a lot of good work on how we teach programming. "Commonsense Computing" [1] found that students understand concurrency a lot faster when presented as a "human" problem, such as selling tickets in a concert. I'd recommend reading Teaching Tech Together (http://teachtogether.tech), which references a lot of empirical papers on teaching programming.

[1]: https://cseweb.ucsd.edu/classes/fa08/cse599/Papers/ICERConcu...

This paper on retros is hilarious gold.

In terms of methodological quality, "Fixing Faults in C and Java Source Code: Abbreviated vs. Full-Word Identifier Names" [0] is a favorite of Hillel Wayne's [1].

> Manageable scope.

> They did their homework.

> They mix qualitative and quantitative methods.

> Objective measure of Defects.

> Really, _really_ good experimental setup.

> And then an ethnography.

[0] http://www2.unibas.it/gscanniello/Giuseppe_Scanniello%40unib...

[1] https://www.hillelwayne.com/post/the-best-se-paper/

I quite like https://neverworkintheory.org/reviews/ which has summaries of recent empirical research papers

Thanks, yes, I put the very interesting

https://neverworkintheory.org/2022/08/30/software-engineerin...

in my newsletter yesterday :-)

A great recap of some papers: Software engineering's greatest hits.

* https://www.youtube.com/watch?v=HrVtA-ue-x0 * slides: https://third-bit.com/talks/greatest-hits/#1

Here's a few I like:

Why Do Computers Stop and What Can Be Done About It? https://pages.cs.wisc.edu/~remzi/Classes/739/Fall2018/Papers...

Not a paper, but the whole book "Accelerate" presents many empirical findings related to automating software operations.

Hidden Technical Debt in Machine Learning Systems - NIPS https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f...

You might like to review the book Evidence-based Software Engineering, which is freely available online: http://www.knosof.co.uk/ESEUR/

Great resource!

This looked promising, while I browsed the index. Especially the first chapters that discuss human cognition. Since software is still a human creative endeavor cognition is key to understand software engineering. However, continuing to read the index I am discouraged and I expect that this is a difficult read. I was expecting more like a handbook, more succint, that I could use to explain software engineering to non technical leaders.

Also, why in the world somebody would choose R as a vehicle to present software engineering fundamentals? This by itself almost disqualifies the book for me. It would have been much more useful, IMO, to use some imperative or object oriented, more common, language.

Mythical man-month - this is a short book written dozens of years ago but from my opinion it is timeless.

Not sure why you were downvoted. Parts of the book are indeed timeless!

My favorite but is the idea that out of the jumble of documents produced, a few emerge as the pivot around which the majority of the team’s communication revolves.

I’ve seen very effective project leads use this to great effect. Rather than robotically using boilerplate document templates they discover the most precise format for the project at hand.

True, but it's not empirical research.

But is that an empirical paper?

On this topic, Peopleware as well. It references a lot of studies and is extremely well researched.

Technically not a paper, but Casey Muratori takes a very empirical approach to software development, which I find extremely practical:

- https://caseymuratori.com/blog_0015

- https://caseymuratori.com/contents

+1 for this. I love the way Casey works

Thanks!

I write and share summaries of papers related to developer productivity here: https://abinoda.substack.com

Two of my favorites so far are https://homepages.dcc.ufmg.br/~figueiredo/disciplinas/papers... and https://faculty.washington.edu/ajko/papers/Li2019WhatDisting...

I have always appreciated https://www.cs.cmu.edu/~NatProg/papers/Ko2008JavaWhyline.pdf - an empirical analysis of debugging and a tool based on those findings, and an evaluation of that tool.

Hard question, there's a lot; one favourite I came across years ago on one uni. course when I was finishing my SW Eng. MSc and while had I worked in Global SW Dev for years. The research was so well put together on a difficult to measure topic:

An empirical study of speed and communication in globally distributed software development

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.324...

The studies referred in this book

https://www.amazon.com/Accelerate-Software-Performing-Techno...

Nicole Forsgren PhD and 2 more Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

Are there studies comparing total development effort to build and maintain software using types vs. no types? You would need to use closely related languages like TypeScript vs. JavaScript, or something like Python with vs. without type hints.

There's some, but it's not... great. The two highest profile studies I know of are:

"A Large-Scale Study of Programming Languages and Code Quality in Github" [1]: Finds significant differences in defect rate between typed and untyped languages. It stood for a while before replicators tore it apart. Among other things, they counted the v8 engine as a javascript project, counted seventeen forks of Bitcoin as distinct projects, and counted commits like "add infix operator" as a sign of a defect (because it has "fix" in it). I wrote a bunch about the study here: https://www.hillelwayne.com/post/this-is-how-science-happens...

"To Type Or Not to Type" [2]: Finds that Typescript catches 15% of bugs missed in Javascript repos. Haven't dug into this as much, but I'm suspicious of it because they didn't seem to do any quality control. One of the codebases they studied, for example, is a junior dev's minesweeper game.[3]

Right now there's a lot of case studies of companies happy with typescript/mypy, which I take as tentative evidence that type systems are on the whole beneficial, but I'm also open to the tide changing in 4-5 years.

[1] https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-...

[2] https://earlbarr.com/publications/typestudy.pdf

[3] https://github.com/anabarasan/Minesweeper

From https://en.wikipedia.org/wiki/Experimental_software_engineer... :

> Experimental software engineering involves running experiments on the processes and procedures involved in the creation of software systems, with the intent that the data be used as the basis of theories about the processes involved in software engineering (theory backed by data is a fundamental tenet of the scientific method). A number of research groups primarily use empirical and experimental techniques.

> The term empirical software engineering emphasizes the use of empirical studies of all kinds to accumulate knowledge. Methods used include experiments, case studies, surveys, and using whatever data is available.

(CS) Papers We Love > https://github.com/papers-we-love/papers-we-love#other-good-... :

- "Systematic Review in Software Engineering" (2005)

-- "The Developed Template for Systematic Reviews in Software Engineering"

- "Happiness and the productivity of software engineers" (2019)

DevTech Research Group (Kibo, Scratch Jr,) > Publications https://sites.bc.edu/devtech/publications/

' > Empirical Research, instruments: https://sites.bc.edu/devtech/about-devtech/empirical-researc...

"SafeScrum: Agile Development of Safety-Critical Software" (2018) > A Summary of Research https://scholar.google.com/scholar?cites=9208467786713301421... (Gscholar features: cited by, Related Articles) https://link.springer.com/chapter/10.1007/978-3-319-99334-8_...

Re: Safety-Critical systems, awesome-safety-critical, and Formal Verification as the ultimate empirical study: https://news.ycombinator.com/item?id=28709239

Can you list the ones you know/found?

You found some in (the very old) "Facts and Fallacies of Software Engineering"

2003 is “very old”? I would have expected that description about something like Structured Programming from 1972. Not 2003.

If not in number of years, more so in half-lifes of knowledge [0].

[0] https://en.wikipedia.org/wiki/Half-life_of_knowledge

"Very old" is subjective, some people think the Antikythera to be very old.

But when I remember developing software in my startup around 2000, the world of software development was completely different. So, hence, for me, very old.

Software is a pop culture.

Reply on news.ycombinator.com