Hacker News Reader

Comment by scott_s | Hacker News Reader

A common thread I'm seeing in the discussion here is confusing the claims from the original paper with conclusions made by this paper. This is a replication study, and it starts by stating what the original paper found. What the original paper found is not what this paper finds. For the most part, this is a replication failure, for a myriad of reasons, including several kinds of misclassification.

From this paper:

The reanalysis failed to validate most of the claims of [18]. As Table 6(d-f) shows, the multiple steps of data cleaning and improved statistical modeling have invalidated the significance of 7 out of 11 languages. Even when the associations are statistically significant, their practical significance is small.

The reference [18] is https://dl.acm.org/citation.cfm?doid=2635868.2635922, which is the original paper that this paper is a replication study of.

View on HN

Yes, definitely. This is mostly a non-replication of the original study. I was a little disappointed that they didn't look at defects per line of code, even though that's already a fairly well-studied metric.

I think the lesson here is that you can write good or bad code in any language, but choose one that's fit for purpose. Using proper abstractions (including those builtin to languages) probably does more to reduce the incidence of bugs than anything you can do other than writing automated tests.

The practice that most reduces the incidence of tests is code review: https://kevin.burke.dev/kevin/the-best-ways-to-find-bugs-in-...

Emphasis on "Formal code review" which is practiced in far fewer shops than "code review".

For those who don't happen to know the difference, formal code review basically uses a check list of common best practices and possible defects to look for versus doing each code review ad-hoc.

I thrive on checklists and hate the ad-hoc way I do code reviews on my team. This sounds great, I'm going to go look for some basic formal checklists. I need language agnostic versions and/or versions for javascript, php, and python, if anyone has suggestions.

The Art of Software Testing by Glenford Myers has an excellent language-agnostic checklist. See the chapter on “walkthroughs”

That is a book that, even in the third edition, has a paragraph that disparages random testing.

It's difficult for me to see how, in the era of jsfunfuzz, Csmith, AFL, and other such tools, such a source should be taken seriously.

A lot of people were right in worthwhile ways and wrong in other ways. For example, Newton.

You would be doing yourself a disservice to throw out all of Newton's results because he didn't grasp relativity.

I’ve never found a book on software I agreed 100% with. Even my favorites have some bits I think are dead wrong. It doesn’t mean the other parts aren’t valuable.

It means the book has not been updated, at least in that part. And the first edition of the book was written back when computing power was seven orders of magnitude more expensive (or more). Is advice about testing from then really going to be valid now?

I think you mean "bugs", not tests.

Sure! thanks for the correction.

So the old adage stands: you can write crap in any language.

And if you do, chances are good it'll end up on GitHub.

5.3 and 5.4 really nail it, IMO.

And you can't even use bug fixes as a metric, because there's a difference between semantic bugs which will be significantly language-influenced, architectural bugs which are partly language-influenced, and user-expectation/experience bugs, which are broad-brush logical errors or misunderstandings.

(There are probably other categories, but those are the obvious ones I can think of right now.)

The real takeaway is there is as yet no objective metric for software quality.

The best a working dev shop can do is implement a small but non-trivial sample project with known-good logic in a handful of candidate languages, and see what kind of trade-offs fall out.

I'll agree with you for sure no objective metric for quality (that is broad enough to capture a satisfying amount of what we mean by quality).

I do think user experience/expectation bugs are very language dependent though. First you have the quality of the String and Datetime libraries. Ruby has all kinds of high quality string functions that are a source of pain and bugs in C.

Say your project wants to display the different between two datetimes as "5 minutes ago" or "last month" (in several languages). Saying "1.27 days ago" is correct to an extent but also not pretty. There exist high quality libraries for this, but not in every language, and a lot of devs would stumble on this feature.

The other is UI creation, I would say it is easier and faster to create a really nice UI with HTML/CSS/JS than with native Android. Maybe I'm wrong on this example but I bet there is a pair of UI frameworks where this is true.

> user-expectation/experience bugs

I consider those to be bugs in the spec, not the code. I once worked at a place that simply swept those under the rug - if it worked as documented, it didn't matter how many users couldn't get it to work. The bug report was closed as "not a bug"; you can imagine how frustrating that was.

tl;dr

Reanalysis showed positive association between defects and C++, negative association for Clojure, Haskell and Ruby.

The original analysis found positive association for C++, Objective-C, C, PHP, Python, JavaScript. Negative association for TypeScript, Clojure, Scala, Haskell, Ruby.

It also found the effect is very small in all cases.

Reply on news.ycombinator.com