Comment by pron | Hacker News Reader

Comment by pron | Hacker News Reader

Parent Full thread

> I fully admit it's hard to convey this and hard to demonstrate this empirically because the cost and scale of an adequate experiment would be prohibitive.

Ah, but you see, here we have a problem. If you claim that your favorite technique has a big measurable impact on software development costs, then observing them shouldn't be hard at all, even without careful empirical studies. There are many problems with free markets, but applying selection pressures that quickly pick adaptive techniques is something markets are actually good at. And if your claim is that the benefit is not easily measurable because it's not so large and decisive, then why is your claim important at all?

And, BTW, I think the preference for a programming style is very personal. I learned and used FP before I learned OOP, and I like both (maybe for different things), but I can't say I have such a strong preference for either. They're both OK.

> but only Coq and Isabelle/HOL is used for the biggest projects like Cake ML and Comp Cert. Both of those are proof theoretic tools.

Coq and Isabelle have so far only been used to verify deep properties of tiny programs, up to ~10KLOC; that's 1/5 of jQuery. Even then the effort was huge. Model checkers have been used on much larger programs, and, at least for small programs, they're commonly and regularly used in industry (especially in safety-critical hard realtime systems).

> If you claim that your favorite technique has a big measurable impact on software development costs, then observing them shouldn't be hard at all, even without careful empirical studies. There are many problems with free markets, but applying selection pressures that quickly pick adaptive techniques is something markets are actually good at.

I'm not convinced this is true in an environment where every problem is unique and failure rates are already high - humans are known to be very bad at working with low probabilities. Even if we're talking about an order-of-magnitude difference, who could distinguish between a startup - or an internal new project - with a 1.5% success rate versus a 15% success rate?

That goes doubly when we're talking about costs: a friend who'd worked for several investment banks observed that they all had roughly the same number of developers overall, but the distribution of those developers was radically different: bank X would have 5 programmers working in area A and 50 programmers working in area B, while bank Y would have the opposite, even though both were solving the same problem in area A and area B. Could you tell whether Google or Facebook has 500 or 5000 or 50000 developers from the outside? Does Uber have 50 or 500 or 5000? (I genuinely don't know).

To my mind the most compelling case: WhatsApp apparently did succeed with an order of magnitude fewer developers than comparable companies - and the industry responded with a collective shrug. (My interpretation is that the industry is still so immature - or so fast-moving - that an order of magnitude cost saving still doesn't matter; software produced at 10x what it "should" cost can still be amply productive/profitable, it's easier to increase profits by increasing revenue than by cutting costs).

> who could distinguish between a startup - or an internal new project - with a 1.5% success rate versus a 15% success rate?

But if we can't distinguish the results, do they really matter? After all, the whole point of changing how we work is to get a measurably better result. If it's not better, why change? (of course, assuming there's a positive effect at all). Something cannot be at once very important and unnoticeable.

> To my mind the most compelling case: WhatsApp apparently did succeed with an order of magnitude fewer developers than comparable companies - and the industry responded with a collective shrug.

A shrug? You can draw a straight line from that to the work I'm doing now in OpenJDK, at the very heart of the mainstream. I know that because WhatsApp's technical success is partly how I sold the idea. Similar efforts are taking place in most other mainstream software platforms. We very much took notice, and we were very much convinced, and that's why we're doing what we're doing. And the contact we've had with the world's biggest software companies and software-dependent companies (including your own employer) about this work shows us that they've taken notice as well.

> But if we can't distinguish the results, do they really matter? After all, the whole point of changing how we work is to get a measurably better result. If it's not better, why change? (of course, assuming there's a positive effect at all). Something cannot be at once very important and unnoticeable.

It can be at once very important and poorly understood. I'd draw an analogy to the state of mathematical analysis in the era of Riemann and Cauchy, where no rigorous basis for the techniques existed; skilled practitioners could and did produce correct theorems, while unskilled students applied the same techniques and produced nonsense. So even when there was ultimately an underlying rigorous reality (as we found out decades later), in practice being able to work effectively was far more a matter of individual flair and taste.

I feel like that's where we are with software development practices at the moment. We all know there's a huge variation in productivity, but to even measure them - much less explain the causal factors - is still a black art. (I'd think the same would be true of artists or research scientists - anyone whose work is fundamentally about original innovation. I'm sure there are techniques and approaches that make an order-of-magnitude difference in productivity, but no agreement about what they are)

Perhaps I'm just more incredulous. Many more people are just as sure that homeopathy works and that vaccines cause autism, so I'd just as soon assume it's psychology at work. But I think what you're saying is that we don't yet have beneficial techniques just ideas that could lead to them. I can't argue with that because I have no information one way or another. All I'm saying is, when someone does figure out how to develop software considerably better/faster, that achievement would be hard to hide.

I suspect that the effective techniques are known (or "known") to some practitioners, but buried among a pile of ineffective techniques, with different people swearing by different techniques based on their experiences. Similar to how traditional medicine can be a mixture of genuinely important treatments for local medical circumstances and completely ineffective treatments that happened to be (spuriously) correlated with good outcomes.

You're quite right about how it looks from the outside; the possibility that I'm just experiencing my own set of spurious correlations is real and worrying. But ultimately there's only so far that I'm willing to overrule my own lived experience.

I wish I shared your confidence about future progress. There have been good and bad managers, and good and bad communicators, for decades, but we don't seem much closer to a generally understood method that can be applied by everyone.

Reply on news.ycombinator.com