Experience report: It will never work in theory
computer.org
computer.org
Culturally, I find we’re simply not motivated by the same things. Researchers want to publish research. I want to ship working software.
The more researchers I’ve met, the more convinced I become that those two goals are at odds with one another.
I suppose your point is that the researcher's goal in not exactly to ship working software either, but wouldn't that put the researcher's goals at worst neutral then?
Likewise your goal is not to publish research, but it is also not to actively work against it either. From their standpoint you're also at worst neutral.
It's also worth pointing out that the motivating factor doesn't necessarily have to be the same for each party to have a common goal. I'd argue that this is how it actually is most of the time.
I do work for a customer so that my boss doesn't fire me and I get my paycheck, whereas my boss does work for the customer to bring money to their company and to not go bankrupt. The customer helps with the work we're doing for them because they want something out of the project that is their money's worth. Our motivations are different, but the goal is the same - to build a thing that works.
My point is that it just seems very likely that some common ground can be found between you and the researchers, regardless of your individual motivations and since there's really no inherent conflict either.
Within organizations, there's an inherent conflict between any orthogonal goals.
In theory there's no conflict, but in practice, there is constant competition for time and resources. This creates conflict between any groups whose goals are not aligned, including groups whose goals are completely unrelated. This is also why organizational politics is the way it is.
I believe these are not - researchers publishing how to create good software and developers creating software may be seen as goals as aligned as fixing an incident and publishing a post mortem, or as writing an RFC and implementing it. Or publishing a post on how you remodeled your infrastructure.
They diverge at some point, yes, but that's not orthogonal at all
I disagree that this is what researchers are doing (at least from my point of view). This is actually an area where I agree with the article, there's a gigantic gulf between researchers and practitioners. The things that academia puts out are not, generally, what I would consider to be good software.
I think this fundamentally comes down to a difference in the definition of "good" between the two camps. So far as I can tell (not being an academic), the academic definition of "good" seems to revolve around software having certain provable characteristics. My definition of good software involves the software exhibiting useful characteristics. And those are, generally speaking, orthogonal. If not sometimes inhibiting each other.
But, of course I would think this, I'm a practitioner. The academics probably have similar complaints about me.
Michael Stonebraker has written/talked about this in the context of DBMS research, but to me the same issues are applicable to other fields as well.
Slides:
http://www.jfsowa.com/ikl/Stonebraker.pdf
Talk:
https://www.youtube.com/watch?v=DJFKl_5JTnA
In short, the incentives/requirements in academia have changed so that producing working software is no longer feasible. He says he got his PhD with exactly 0 publications, became a professor with 5.
Nowadays, you have to have 5 publications to get your PhD and god knows how many to become professor.
That means 1 paper successfully published per year if you want to get your PhD in 5 years. Doing software is not possible, it is too risky and just takes too long.
There's a lot more, I highly recommend the talk!
Why is it a problem they are at odds? Its not a researchers problem to create scalable production/consumer software or hell even git repos or libraries for you. Its yours! Thats your interest.
Doing applied research and working with people who want to do that tends to work reasonably well with production, because the point is to research techniques and algorithms and so forth that help improve production. Theoretical or more future-focused research tends to need to focus on building minimal scaffolding to be able to test ideas, when that kind of scaffolding gets integrated into production software, that’s where the problems arise, because by necessity it’s a kind of MVP. I am right now working on rearchitecting a component that was built this way and labeled as production without going through hardening first (though it did go through a great deal of testing it didn’t get the code review it needed).
In the end I suppose my point is that research and production aren’t inherently at odds, or even orthogonal. Applied research can be valuable to improving production software by studying current stable code and processes and researching improvements, but trying to do blue-sky research by incorporating code into production can be orthogonal or in some cases truly at odds with stability. It matters a great deal who is doing it, and how it’s managed.
[1]: https://a16zcrypto.com/posts/article/accelerating-the-world-...
[2]: https://a16zcrypto.com/posts/article/a-new-era-in-snark-desi...
Based on ample experience, both first and second hand, that's going to fail. It is really hard to instill proper methodology, domain modelling, and statistics in students, and since the premise already is that it "would not disrupt other curricula" (PM speak for: little time and attention), students are going to walk away with the wrong ideas and think that they now understand experimental science.
Unfortunately, most researchers are publishing work about improving process, creating new and interesting things, and developing new ways of thinking about the science of computation, instead of what actually makes money, throwing out half-baked apps faster. There’s really no need for a formal understanding of what you’re actually doing and why if you can throw out a few trial-and-error projects and one gains enough traction.
edit: On another level I am put off by their tendency to describe things such as "teaching undergraduates" or "debugging problems" as algorithms. I understand that you can't really analyze unclear, hazy things without putting them down in clear terms, but something in me doesn't really accept that you _can_ do this without losing so much detail and information as to make your conclusion useless.
They do point out that SWE often think of studies this way:
> Most software developers in industry have never heard of any findings more recent than The Mythical Man-Month: Essays on Software Engineering (which few of them ever actually read) and routinely dismiss studies as “not statistically significant,” even when those studies are carefully done and directly relevant to their work.
I love the premise but find the recommended results silly. TL;DW: cognitive research shows humans have working memory limited to 7 +/- 2 things. This means that classes should be limited in what they do. So far, I agree. And I think this can be extended, in practice.
However, they back up this idea with a single case study and use a decorator to help count complexity to keep the classes small and state that methods not extending beyond 24 lines of code are ways to do this. This will just lead to cargo culture like "don't comment code."
The decorator is not needed because code should (authoring or in review) be obvious if handling more than 5 to 9 things. The number of lines might be correlated to the number of things a function does, but that doesn't mean cognitive overhead. Like, a 100 line function that maps state abbreviations to state names is fine.
Based on the presented data and mental performative drop off, it is not 7 +/- 2, it is really five + 0 to 2. I hereby dub this the Rule of Five.
A practical example better than what they present in the talk: early returns. Every nested block adds to cognitive load, early returns allow you to discard that load as you continue reading through the function. The cyclomatic complexity reaching over 5 levels means many people will start to fail to keep all the blocks in working human memory. Therefore, early returns are a good implementation of Cognitive Driven Development.
The case study they used seems meh. With a couple decades of experience with high growth start ups becoming "real" companies, I just don't buy their results on decorators and function line length. But they are on to something with limiting code to keep minimal cognitive overload down.
So yeah, in practice, I agree to keep classes small, but not for the short term memory problem.
A silly example: when you are learning to dress yourself as a kid, zippers are hard, shoe laces are hard, putting left shoe on left food often gets mixed up. All these little details. When you are older, these details take no mental capacity and you can "get dressed" as a single concept.
Living like this sounds exhausting. The only thing worse I can imagine is having to listen to people who actually think like this.
The founder posted on their blog about how every year has things going on, and they still need to raise money. And the users completely backlash against all forms of monetization.
What I didn’t see anyone mention is the users are still using the site. Apparently using the site is really important. Just not making it, or earning a living while making it.
It's like someone took the CIA handbook on how to sabotage institutions and used it on every organization more left wing than the John Birch Society.
In general, practitioners would be interested in hearing from advances in academia, but usually don't want to read papers. Building fully working systems - programming languages or libraries - is more impactful, but much higher-effort.
Isn't that kind of the whole problem? Practitioners ignoring research results because all they want to look at is what other practitioners do? Or are you really suggesting that researchers also become practitioners who develop working systems based on the ideas they discover?
Perhaps I'm misreading, but your statement seems to encapsulate the entire problem the article is highlighting.
Of course in this publish-or-perish era, this is probably no longer possible.
(... was back then, and I was amazed, of course.)
I know this is really just a turn of phrase, but at the rate language models are becoming programming models, it is hard to think of any problem with the lack of hard engineering, science, math, ..., in what we call "Computer Engineering", "Computer Science", "Computer Math", ..., that won't be heavily impacted by model/co-model driven development.
For instance, a major programming language of the near future might develop adaptively as multiple coder models communicate between each other. Reviewing, testing and improving upon each other's designs and co-optimizing the way they communicate code as they do so. Resulting in representations and solutions to existing development problems completely opaque to the humans currently working on those problems.
The rising effect of massive data + compute outdoing slow careful research in yet another area.
Today's models are already being trained across the board on practical/actual and theoretical/research artifacts. No persistent pernicious borders there.
Not to take away from anything being said.
But why would anyone want that? I agree that this would result, but my takeaway is "and therefore we shouldn't do that".
My guess is that a lot of CEO's already consider the implementation details of their companies' software systems as opaque. So that is essentially a pre-existing market for performative but opaque systems. CEO's. CFO's. Shareholders, etc.
Even in engineering organizations. Look at how Boeing and NASA both demoted critical engineering concerns when external concerns became overwhelmingly more important to their leaders.
Now factor in the models actually doing a good job, checking each other, and generally getting better year to year.
To be fair, that does exclude quite a lot of problems.
> To be fair, that does exclude quite a lot of problems.
I had some trouble parsing the comment, but I eventually settled on the reading that it means what it literally says, a problem caused by the lack of engineering in computer engineering, of science in computer science, and of math in "computer math" (a phrase I have never heard), rather than a problem other than hard engineering, science, and math, which I agree with you excludes an awful lot of problems.
My first computing programming class was called Computer Math. With not even a soft math emphasis.