The underwhelming impact of software engineering research
iansommerville.com
iansommerville.com
It's crazy to me that researchers prefer to be published over having world renowned software to their name. You could figure out the fanciest database research, but there was a period where every fancy new NoSQL database was based on BerkeleyDB despite all that, bolstering the reputation of the University of Berkeley.
Despite all the research into garbage collectors nearly every new programming language I've seen uses either BoehmGC (from 1994) or simply gives up on the idea and uses reference counting.
Imagine if university groups would band together to produce really state of the art production quality reference software. They could use them as base for their research, bolster their reputation by having them be great, and even draw funding from big companies looking to hire the PhD's produced by the efforts.
Instead we have got all these half finished demonstration software that no one industry would touch with a ten foot pole.
It took just a few people to develop the first versions of rust, a research based programming language that in my opinion has been the biggest shock to the industry in decades (if you disregard machine learning).
Haskell even has problems coming up with a single string / text interface. It's not some magic bullet for code reuse, any more "everybody would write Java and we would have had massive amounts of reusable jar libs".
That's orthogonal to the claim that if everyone used Haskell we'd have better code reuse.
> It's not some magic bullet for code reuse
Not a magic bullet, but makes code reuse much easier than any other lanugage.
That can be attributed to a focus on sound abstractions, referential transparency, and laziness[0] mean that code is much easier to reuse.
0: Yes, laziness helps composition and code reuse!
It’s much more lucrative to be a firewoman when there’s idiots pouring gasoline and throwing matches everywhere.
Unless the BerkeleyDB was written based on novel research, it didn't move the state of the art forward, it was just a good, used, embeded database.
In which case, your statement can be rewritten as:
"It's crazy to me that researchers prefer to be known for their research over being known as developers of popular programs".
Doing research is the reason they went to an academic career, if they wanted to be developers, they'd just have gone and be developers.
So, why not strike two birds with one stone, and make high quality widely used software that's ALSO state of the art research based?
Well, because creating "high quality widely used software" is a different process, and needs different resources, dedication, and even skills, than doing research.
To showcase some novel research in practice is much much faster, easier, and cheaper to create "half finished demonstration software", that to create a mature, final, implementation that people can use.
So, either they slow their research considerably, and spent most of the time doing software development work - which besides slowing down the pace of research will also count against them in their academic field, they opt to focus on research and just do simple implementation that are not production ready and aren't meant to be.
Seen in this light, it's not a "crazy" choice, it's an essential tradeoff.
While it has some great benefits once you arrive, the upfront costs are enormous. You need to find a funding source (or sources) that will pay for this product while you're building it. Also, in order for the research payoff to be worth it, you need both the product itself, and subsequent innovations it enables, to be research-worthy. Not all areas of research can support this. On top of it all, even when you do this, you'll still spend years of effort in activities that are essentially not research. You're basically responsible for all of your own customer support, sales, marketing, etc.---like a startup, but without the financial upside if you succeed. Yes there is recognition and so on, but the payoffs aren't as dramatic. Most people aren't ready to commit to this path.
Keep in mind that you can't build this in 5 years either. So a single generation of PhD students can't get it done. The only reason we were successful is because the key staff on the project stuck around for 5+ years after our PhDs because we all believed in doing the work.
Given all that, I don't hold it against people at all who just want to build prototypes and then move on to the next thing. It's way less risky and higher reward relative to the costs.
[1]: https://legion.stanford.edu/
[2]: https://flexflow.ai/
If you went into research, you have occupational incentives to publish papers, but not to publish software.
As someone who went into applied software, there's a lot of published research that's so disconnected from reality, that it couldn't possibly be useful for anything but publication stats for the named researchers. On the other hand, every once in a while, there's a gem that's useful.
My personal favorite is network research where everything was done with models of TCP that aren't at all similar to actual TCP stacks, production or research.
There is nothing wrong with solving old problems: that describes most useful human work. It’s just that finding and solving new problems is a very specialized kind of work, and it’s very hard to do that specialist work if you aren’t spending most of your time on it. You need to keep up with the literature and fight to stay at the forefront of your field and you can’t do that as a 20% time position.
A good analogy here is like the interplay between a general contractor and a licensed electrician. The contractor could do their own wiring in between framing and drywall (and take the necessary tests and certifications to learn the code), but it usually makes more sense to have specialized contractors come in for that work. Unfortunately the problem described by this article is that the software developers (contractors) aren’t using the output of the researchers. Insofar as this is true it should be addressed. However the answer probably is not to have the specialized researchers do more general software development, which will just make everyone less efficient.
As far as I remember, the story went something like this, about ten years ago:
- organisation employing a PostgreSQL developer asks "can we have SERIALIZABLE that actually does what it says?"
- developer says "nobody has that, it's a hard research problem"
- developer looks for recent academic research and finds someone with a proposal that might be efficient enough to be practical
- developer contacts the researcher
- developer and researcher implement the proposal in PostgreSQL
- it turns out the design is in fact practical
- PostgreSQL gets a useful feature, researcher gets a paper with the developer as co-author
Looks like this is the paper: https://pmg.csail.mit.edu/pubs/ports12__serial_snaps_isolat_...
What this means is that academic software projects become spaghetti code messes that no one wants to touch at warp speed.
Have you ever seen a garbage collector that uses the NSA instruction? https://github.com/jart/cosmopolitan/tree/master/tool/plinko
From my experience with building my own GC. There are only two challenges. Stack unwinding to obtain root references and then the GC strategy (generational GC, mark and sweep, etc).
For most people, reimplementing the latter is a waste of their time.
A drop in garbage collector (like Boehm) makes your toy language instantly production ready. Basically code generation and garbage collection are the two things that make language implementation tedious. That's why so many languages are on top of Java or .Net. Implementing a language on those platforms is almost trivial because it solves both code generation and garbage collection.
What new language implementations are using Boehm?
You use Java's GC right? Or is there something custom in there now?
Most new languages are either high level languages and running on the JVM, or they're attempts to compete with C/C++ like Rust, Swift, Zig etc and thus don't use GC at all. I don't think many people use Boehm anymore.
In a world where the dependent variable isn't something that can easily be measured in a lab, the next best we can do is either 1\ Ask experts or 2\ experiment in an environment where it's financially advisable to experiment & measure (AKA an environment where running an experiment with 100 engineers is <1% of your engineering headcount).
I.e. I don't work for Google, but I do share things like [0, 1] with new engineers who join my team.
[0] https://google.github.io/eng-practices/ [1] https://arxiv.org/ftp/arxiv/papers/1702/1702.01715.pdf
Beyond that, a lot of software engineering research doesn't seem to be easily reachable or discovered unless you're in academia.
Real world software engineering has to ship a real working product, while academic software engineering only has to produce a paper. Maybe that's an unfair representation though.
Building these products would be a great teaching tool and contribution to society. Also there could be a lot of research done on how to build complex software where the product is the software itself and it can be applied to other sciences that badly need software. I for one would love to get a graduate degree while building some product where I get to try different software design approaches.
There are any number of reasons I have stayed engineering track instead of going into management but the biggest is that talking about things and doing them are vastly different, and we’ve structured management and research around talking. Some days I worry I may know more about project management than I know about cloud or React, but it would kill me to be telling people how to do the job that I’m not also doing, and if I don’t like how they did it, have only words available to get it fixed.
In one of my hobbies, time and the pandemic have slowed me down quite a bit and I’m trying to manage more and do less, but unlike work there are plenty of opportunities to lead by doing. Doing 5-10% of the work and then guiding people to do the rest is socially acceptable.
In software we sometimes call that a lead developer, but that definition changes from situation to situation. But the more you do the less influence you tend to have on what you’re doing and why. Is that human nature or leftover from feudalism?
It's funny, back in my undergrad days I would walk past the Google offices that opened near my campus. I'd look in and see a giant wall-sized candy machine, primary colors everywhere, bean bags, pool tables, an air hockey table. I honestly thought it was a daycare of some kind until I asked someone why there's never any kids in there. This is what they call the "real world" apparently, as a contrast to what goes on in academia.
I can definitely say that in all my years in academia, I've never been treated to a candy wall.
It’s clear that there’s been lots of progress in technology: in networking, in databases, in compilers, etc. It’s just that none of that was substantially influenced by research on the process of software engineering.
Applying the results of research on these kind of practices can be incredibly difficult. Engineering processes are not a hard science…
Hardware still works this way today basically. The performance optimizations, practices and goals are engineering centric, not consumer centric. Consumer benefits happen downstream as a side effect. The hardware sector is still run by scientists and engineers, one only needs to look at the people running the companies.
There's no shortage of great software research that could be put into practice, plenty of languages with great features are underused. It's the result of the mentality in software that puts costs, users and marketshare over an engineering culture.
Few software developers are required to acquire professional certification, or perform in compliance to standard practices intended to assure due diligence in design, implementation, testing or documentation. Very few software systems must undergo compliance certification. It's unsurprising then that software engineering continues to founder and fail to advance as a discipline.
1. The widespread adoption of unit testing and associated techniques like continuous integration. A massive boost to software engineering reliability and productivity.
2. Introduction of crash reporters and associated analytics, this enabled huge improvements in software stability. On the server side, good logging and monitoring systems provide an equivalent.
3. Sandboxing going from unheard of to relatively widely used.
4. Design, development and then open sourcing of extremely robust and well engineered runtimes like the JVM, .NET etc. This has huge benefits throughout the entire industry.
5. Open source and the huge code reuse it made possible.
etc etc. All these things have been drastically more impactful than bureaucratic central planning a la certifications.
Compilers that work like devs do. Almost every time I compile, the code is almost exactly the same, but the compiler forgets everything and throws away details from the last compile. Probably all the optimizer's choices are going to be the same this time as last time, so maybe we could cache these?
Better code merging. When my code diverges from the main branch, I typically have to do a bunch of merging work. Would a merging algorithm that understands code semantics be able to do a better job of understanding what I mean during the merge?
Language safety and ergonomics is huge. I very much appreciate Rust's progress, and it's a shining example of academia brought to industry, but can we do more? Jai's idea of making constraints codeable and incremental is pretty cool, and I bet there's a lot of room to improve on it.
Measures of code complexity and modularity. There's a bunch of silly measurements of the "goodness" of code, but maybe it's time for another! Maybe we could describe how "replaceable" components are automatically, and gives a solid idea of the level of real tech debt in the codebase.
How about some project management focused studies? Have a programmer give an estimate for a task, try to force it to be a deadline, and watch as quality of the results drops. Try applying something like scaled agile framework to a real problem and watch as the meeting load snuffs out meaningful development progress.
Or some systems work. Apache airflow is a pretty cool job running agent, but I have yet to see a good error handling workflow for distributed job agents. Can you come up with the "Result" type of job scheduling that's nicely composable?
There's really so much to do.
The tradeoff is that people need to write some code to tell the VCS about the format of each binary file type or semantics of each programming language.
The biggest problem is that, like Rust, a new VCS has to be well-executed to make its innovation stick. We'll see if I succeed.
[1]: https://tahoe-lafs.org/~zooko/badmerge/simple.html
[2]: https://tahoe-lafs.org/~zooko/badmerge/concrete-bad-semantic...
So, this criticism resonates a bit with me. The problem with software engineering as a scientific discipline is that many engineers probably would not consider it a proper engineering discipline to begin with. Also, many computer scientists look down on it as not really related to that either because it is a bit of a soft science.
So, there's a problem with researchers in this field not really having the right background to do good research:
- they are typically not experienced software engineers. Quite many, like me, start working on a PhD without actually having worked as a software engineer.
- they typically have studied computer science which teaches you a lot of math, algorithms, formal methods, and other typical beta skills but is perhaps a bit light on the type of stuff a sociologist or anthropologist would learn. That's a problem because a lot of software engineering involves the behavior of people in groups and coordinating that behavior to produce software. Studying that behavior to verify theories you might have about how software engineering works or ought to work requires an empirical approach. That's not something a lot of computer scientists learn how to do.
So there's a lot of software engineering research that gets published that you could accuse of being a combination of naive, wrong/misguided, and poorly executed in terms of its methodology. Mathematicians or computer scientists trying to be empirical is not great typically. Sometimes you get some decent papers but there is a lot of pseudo scientific nonsense that gets published. Doing empirical research properly is hard and many computer scientists have no clue how to do it properly nor any appreciation for doing so.
Just looking at ICSE's program last year you can see from the titles a weird mix of empirical studies of various topics and other papers clearly written by more computer science leaning types working on algorithms, machine learning or tools. Looks like there were some interesting papers in there but also a lot eye brow raising titles like "How Java Programmers Test Exceptional Behavior". I actually skimmed through the pdf of that one and it fits the pattern beautifully. It's an actual empirical study that tries to answer the question in the title. That's it. There's no new theories being formulated. No remarkable findings that were reported. It's just a very dry survey that yielded exactly what you'd expect. The answers to the research question are almost beside the point and should be completely unsurprising to any Java programmer. Great example of a well executed emprical research paper that answers a question any Java developer in need of actually testing some code that throws exceptions would answer with a simple Google query in a few minutes.
So, yes underwhelming impact indeed.
I'd hypothesize that most applicable improvements to a software engineering company are about - collaboration & best practices - than anything highly technical.
Calling software engineering "engineering" is perhaps the most blatant uses of describing a thing by it's exact opposite. Like calling hate love, or war piece. (Which sometimes is done, but mainly for propaganda purposes - for the intent of obfuscation.)
Software engineering is much closer to a craft than anything capable of teaching or predicting anything successfully. The best analogue that comes to my mind are the craftsmen who built cathedrals in medieval Europe. Yes, they built things. But they did not understand with scientific rigor, what and why are they doing it. They did not have proper theory, so they could have built something completely different.
You can learn the activity known as software engineering. For most individuals, the best way is from an informal apprenticeship by working with talented colleagues. Again, this is closer to art or a craft than engineering.
You can learn the basics in structured setting. But based on the stories of long winded recruiting interviews, nobody actually trusts any of that, or at least they don't trust the accreditation mechanism. Which implies you can pass a degree, but only accidentally learn the skills the industry expects a great candidate to have.
Something is institutionally broken here. Imagine if medicine worked this way. Are doctors whiteboarded over complex surgeries before being hired?
I would consider the work done by NASA's software engineering laboratory one of the things closest to "proper engineering" done in the field of software http://www.cs.umd.edu/projects/SoftEng/ESEG/papers/83.88.pdf
Is there anything similar currently operating?
One thing that particularly scratches me lately is the popularity of Actor-Network-Theory in some of the humanities: It is a theory that might benefit quite a lot from CS/Maths input, and I think it could maybe be applied quite well in SWE theory as well.
I have no idea if it is actually the case, but one thing I'm quite sure of at this point is that humanities need more formal systems, and SWE research needs to look more towards the humanities and social sciences in terms of frameworks to explain human and machine interaction.
Sounds like “capitalism” became the philosophy that drives our research nowadays and researchers optimize for it.
I am really pleased to hear a heavyweight like Ian Sommerville pitching in on what I've been seeing as a cultural crisis in software spanning almost a couple of decades now. It's disappointing to hear the usual dismissal of research and lionising of "industry" as if a cowboy "move fast and break things" mind-set constituted a single coherent commercial ideology. Can you imagine if the pharmaceutical or aviation industry took a cavalier attitude towards "eggheads with their cell biology theories" or "crazy academics with their fancy aerodynamic equations".
Is software the only engineering discipline where the product gets less efficient each year? Partly I think it's also the public's fault for accepting such shoddy produce, and of governments for not being stricter because of a Faustian bargain to enjoy mass surveillance and cybernetic social control as vicarious side effects of a hands-off approach.
It must be frustrating to have put a lifetime into carefully considered formalisms and see so little uptake.
Anyway, what we've ended up with a bunch of non-interoperable, insecure devices and services built by well paid but mostly unhappy workers who have almost no grasp of the complexity, history or societal import of what they create. Sooner or later the chickens are coming home to roost, as cybersecurity, resilience or interoperability issues.
Says Sommerville:
"As a community, senior members (and in this I include myself) have failed to create an inclusive environment where new ideas from outside of the traditional elite are supported. We do not act together when dealing with policy makers and funding agencies and, with rare exceptions, have not spoken out against the futile short-termism that has hampered the development of our field."
I absolutely salute you Ian. I wrote similarly a year ago in my last book that I feel a great sense of regret as a computer scientist who did not speak out sooner and louder. Perhaps it's grandiose/omnipotent to think we ever could have made a difference, but if not our generation then I really hope a new, radical generation will emerge who have the courage to "take back tech".
There's been considerable progress in programming languages and tools. It's now possible to write very large programs with extensive concurrency without having them crash frequently.
There's been progress in project organization tools. There's Github, and Slack, and Stack Overflow. It's now possible for a large number of people to collaborate on something and still get it to work.
That's pretty good, actually.
Does this stuff come from academia? That's what's being debated.
Or GitHub copilot?