Hacker News Reader

Peter Norvig made a comment here that agrees with you about process and architecture over language (he was talking about why he started using python instead of lisp)

"In terms of programming-in-the-large, at Google and elsewhere, I think that language choice is not as important as all the other choices: if you have the right overall architecture, the right team of programmers, the right development process that allows for rapid development with continuous improvement, then many languages will work for you; if you don't have those things you're in trouble regardless of your language choice."

https://news.ycombinator.com/item?id=1803815

If we're going to talk about empirical effects, we first need to agree on what quality means. Parts of the thread, including your comment, lead me to believe we sometimes mean an abstract, desirable but ineffable property--but then we're not speaking empirically.

Do you mean "defect rate" or something else?

I mean any reasonable metric you pick. The studies that have tried and failed to find a big effect used defect rate as a metric, but studies are not the only way to detect big effects. Because there are strong selective pressures in industry that can detect big effects, we can say that no big effect on the bottom line has been found.

> then the reality is that no big effect has been found for the choice of programming language on software quality

Substantiate that. I say it's garbage.

<https://www.cse.iitk.ac.in/users/karkare/courses/2010/cs653/..., and this BTW is from 1994 so this evidence has been around for ages

  Language LOC Documentation Lines Dev time (hours)
  (1)Haskell 85  465  10
  (2)Ada 767  714  23
  (3)Ada9X 800  200  28
  (4)C++ 1105  130  –
  (5)Awk/Nawk 250  150  –
  (6)Rapide 157  0  54
  (7)Griffin 251  0  34
  (8)Proteus 293  79  26
  (9)RelationalLisp 274  12  3
 (10)Haskell 156  112  8

(Edit: all aligned with tabs but looks like they've been stripped. See paper instead)

comments of note:

(2) The Ada solution was written by a lead programmer at NSWC; in this sense it represents the “control” group. The developer initially reported a line count of only 249; this was the numberof lines of imperativestatements as reported by theSun Adacompiler, anddid notinclude declarations, all of which were essential for proper execution. The line count of 767 is based on the actual code in and does not include lines with only termination characters.

(4) TheC++solutionwaswrittenbyanONRprogrammanagerafterhavingfirstwrittentheAwk solution described in the next paragraph. In addition to these 1105 lines of code, the developer also wrote a 595-line “test harness.” No development times were reported [Note: C++ has evolved a lot since then, but perhaps so has haskell]

(10) Intermetrics, independently and without the knowledge of NSWC or Yale University, con-ductedan experimentof its own: theHaskell Report was given to anewly hiredcollege graduate, who was then given 8 days to learn Haskell. This new hire received no formal training, but was allowed to ask an experienced Haskell programmer questions as issues came up during the self study. After this training period, the new hire was handed the geo-server specification and asked to write aprototype in Haskell. Theresulting metrics shown in row 10 ofthe tableare perhaps the most stunning of the lot, suggesting the ease with which Haskell may be learned and effectively utilized.

Also mentioned was "The use of higher-order functions [in haskell] is noteworthy" so language features did help. Actual evidence.

> As lack of evidence in favor of a big effect is evidence of its lack

yeah right. Let me rephrase that for you - Lack of knowledge by an HN poster is not evidence of lack of evidence.

> What has been shown to drive quality up or down is process

Evidence please? My experience is that process can be rigorous but useless crap. My old boss said "process is not sufficient to produce quality, but it is a necessary"

There's plenty more in that PDF, please read it.

> this evidence has been around for ages

Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more than a few hundred lines. I'm talking about effects on software development. The issue of prototyping/tiny programs is a completely separate one.

> Substantiate that

I don't know how I can substantiate the lack of evidence, but here's a recent failed attempt to find evidence: https://arxiv.org/pdf/1901.10220.pdf

> Lack of knowledge by an HN poster is not evidence of lack of evidence.

True. While I have been following the subject for at least the past 20 years (and I have read your PDF several times already, including shortly after it was published), it is certainly possible I have missed something. If there's any evidence of a large effect you believe you've found, let me know.

> Evidence please?

Sure. Here's code review, for example (I don't have time to discuss each paper -- as they're quite different -- but they all paint a similar picture). BTW, while programming language studies are debating effects in the 0-15% range, these report effects in the 30-80%:

* What We Have Learned About Fighting Defects, 2002 -- https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf

* The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data, 2009 -- https://www.pitt.edu/~ckemerer/PSP_Data.pdf

* Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software, 2017 -- https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-...

* What Types of Defects Are Really Discovered in Code Reviews?, 2003 -- https://ieeexplore.ieee.org/document/4604671

* Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?, 2014 -- https://www.testroots.org/assets/papers/2014_beller_bacchell...

* The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects, 2014 -- https://dl.acm.org/citation.cfm?id=2597076

* Best Kept Secrets of Peer Code Review, 2013 -- https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets...

* Modern Code Review: A Case Study at Google, 2018 -- https://storage.googleapis.com/pub-tools-public-publication-...

If we summarize our current state of knowledge (not myth) it is this: process matters a lot; language matters little, if at all (with all the caveats I mentioned in other comments, such as diminishing returns etc.).

You're moving goalposts. You denied language mattered:

> then the reality is that no big effect has been found for the choice of programming language on software quality

Then I show some evidence giving small programs rapidly developed, which you then say doesn't count because it's 'tiny' and 'prototyping'. You also don't define quality which gives you lots of wriggle room.

Nope, 1000 line programs aren't tiny. They aren't industry monsters but you can't dismiss them because it contradicts you. It's not proof, but it is strong evidence.

...arxiv paper... Interesting, thanks. It would have been helpful to have posted that in your original post.

However from <https://soarsmu.github.io/papers/A_Large_Scale_Study_of_Mult..., and note this paper is not mentioned, as in not debunked, in the above arxiv paper.

"Bhattacharya et al. study four open-source projects which use C and C++, i.e., Firefox, Blender, VLC Media Player and MySQL to understand the impact of languages on software quality [20]. They compute several statistical measures while controlling for factors, such as developer competence and software process. They find that applications previously written in C are migrating to C++ and C++ code is often of higher quality, less prone to bugs, and easier to maintain than C code."

and from that same paper:

"As can be seen in Tables 8, 9, and 10, the mean de- fect density values for the C sets can be up to an order of magnitude higher than the mean values for the C++ sets."

> Sure. Here's code review...

I was talking about your claim about languages not mattering, I did not mention code review. All these papers are about code review. This is relevant to my code review comment (edit: I meant process comment), they don't say anything about language vs bugs (unless you wish to point out a paper that does).

> process matters a lot

And I agreed with you, I just said it didn't deliver quality, it just prepared the ground for it. With enough process you can close any hole in a language, but it gets exponentially expensive.

> You're moving goalposts.

Only if you insist on uncharitable reading. I'm talking about software development; if I mistakenly assumed that could be left implicit, I'm sorry. And ~500 line programs are positively minuscule. JQuery is ~50KLOC, and an average business system is ~5MLOC. We're talking four orders of magnitude between those prototypes and a rather average industry system size (systems commonly run to tens and even more than 100 MLOC).

> It's not proof, but it is strong evidence.

I don't think it matters if it's strong or weak evidence, as it's not even about software development.

> the C sets can be up to an order of magnitude higher than the mean values for the C++ sets.

As I mentioned in another comment, C is not a good example. It's a ~50-year-old language, and the theory that correctly predicted that languages won't make a big difference was based on diminishing returns. I.e. not that no two languages have ever been or could ever have a big difference, but that over time the ability to affect quality with language would severely diminish. The original prediction of that theory was that no 10x improvement would be made by a single language improvement over a decade, and was called overly pessimistic by PL fans. It's not been over thirty years, and we have doubtfully made a 3x boost with all language features combined.

If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric.

> I just said it didn't deliver quality

But those papers show that, unlike language, process does have a big impact on quality.

> With enough process you can close any hole in a language, but it gets exponentially expensive.

You are now making an unsubstantiated claim that contradicts a substantiated claim. Programming languages (with the caveats above) have not been found to have a big effect (nor is there a theory that suggests they do), while process has.

> I'm talking about software development

And the value different languages may add is entirely about software development.

> I don't think it matters if it's strong or weak evidence, as it's not even about software development.

Programming languages are not about software development? Oh do go on.

> As I mentioned in another comment, C is not a good example.

A study of C vs C++ strongly indicates your claim about languages is false and suddenly C is not a good example? I'm trying to believe I'm just misunderstanding you but it's getting more difficult.

> The original prediction of that theory was that no 10x improvement would be made by a single language improvement

This isn't what you said. Let me remind you "If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality"

The goalposts aren't being moved, you've just bought them plane tickets to barbados.

> If you want a more precise statement: no theory or empirical evidence supports the claim that a reasonable choice among production languages developed in the past three decades or so has a big impact on any measurable bottom-line metric

Another claim. Show me the study that says that.

> But those papers show that, unlike language, process does have a big impact on quality.

It can if done properly; it does not do so automatically. You can have code reviews that are of little use because you have no spec to review the code against. I have been in that very position. I don't dispute the value of process but it's a necessary but not sufficient condition to bring about quality. I also agree code reviews are good, if done properly.

ME >> With enough process you can close any hole in a language, but it gets exponentially expensive.

YOU > You are now making an unsubstantiated claim that contradicts a substantiated claim.

If I use C I have to worry about garbage leaks, double-freeing pointers, out-of-bounds accesses etc that can be detected with code reviews. If I use python, I never have any of these problems so a code review need not check for these. That's a lot cheaper.

(Edit: weird stuff happening to my post, may appear twice)

> This isn't what you said.

I can't repeat all the caveats every time I say something. I assume readers believe my comments are at least reasonable, even if they disagree with them. For convenience, I restated the current state of knowledge more precisely in my previous comment. If you're interested in what I have to say, assume I'm not an idiot, and perhaps we could have an interesting discussion, and if you think I'm an idiot, then there's no point in arguing at all.

> Show me the study that says that.

Show me the study that says otherwise. I don't need to provide evidence for the lack of an effect (although I did). The lack of an effect is, at the very least, the default hypothesis in this case (as in many others) as there is no theory suggesting we should believe otherwise (while a theory that explains why languages increasingly have smaller effects has made correct predictions).

> It can if done properly

That's your hypothesis. We've been trying to find evidence of that, or others like it, for a long time -- both in academia and in industry -- without success. We simply do not observe that the choice of language today (among reasonable ones etc.) has a big effect, either in studies or in industry practice. But if you've found a language that can drastically reduce software development costs and/or increase quality at scale, that discovery can be easily translated to billions of dollars. Go ahead and make them.