Comment by robbrit | Hacker News Reader

Comment by robbrit | Hacker News Reader

Seems like they're just doing associations, and not controlling for a lot of factors. For example, you can come up with some narratives to explain a lot of these findings:

* Python, JS, PHP, Java, C++ are all very common first languages; I'd imagine the average experience level for these is dragged down a bit compared to others like Golang, Haskell, or Scala. One of the recent Stack Overflow Developer surveys found Golang use to be strongly correlated with age, and since I'm assuming bug count would negatively correlate with age you'd see a confounding factor here.

* Ruby's community has an obsessive culture towards unit testing, that could impact bug levels.

* The trade-offs that make C and C++ attractive such as memory performance are double-edged swords and will naturally correlate with more bugs. For example, I work a fair bit in embedded systems which use MISRA C out of necessity; I've also worked in large scale data processing which chooses C++ for its memory mapping performance over Java or Go. If "number of defects" is your primary measure then you're ignoring some of the reasons languages are chosen. Sometimes a language that minimizes bugs is not a viable option.

* I'd guess there is a skill correlation here: an engineer who is willing to seek out a specific less-used language and learn to use it effectively is probably more likely (not guaranteed) to do a better job than one who is just going with the default choice.

It'd be very interesting to see a regression done that accounts for these factors.

I think you're making the same confusion a lot of other commenters have made, specifically responding to the original study as opposed to this failed replication of that study: https://news.ycombinator.com/item?id=20105150

Ruby's community has an obsessive culture towards unit testing, that could impact bug levels.

Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation. So then, could the argument be made that less stringent type annotation hurts code quality by removing effort from unit testing?

> they're forced to do this

Figuratively speaking. None of the languages they test enforce any unit testing. What I'm suggesting is that since the Ruby community tends to be more obsessive about testing, that might lead to an improvement in code quality when comparing against other similar languages like PHP or JS which are less obsessive about testing. It's something you might have to control for to see the impact.

Type systems are a completely different factor; you'd have to control for that variable too to determine the impact of it. My experience with both systems makes me predict that static typing would improve code quality, but if you want to actually be scientific you'd want to add that factor to the statistical analysis to see if the data support that hypothesis. Anecdotes like personal experience don't count.

Figuratively speaking. None of the languages they test enforce any unit testing.

The trivial type-mistake "Unhandled Exceptions" in Smalltalk would leak out more often than was comfortable, if the project didn't have a good means of testing to catch those. I suspect that has something to do with the Ruby community being more obsessive about testing.

> "Unhandled Exceptions" in Smalltalk

?

MessageNotUnderstood

Right. This, and many others, would result in an "Unhandled Exception" notifier dialog in many versions of Smalltalk. I used to work for the company acting as the vendor of 2 (3?) commercial Smalltalks, also doing consulting work for others.

No, the behavioral test [1] one writes cannot be replaced by type checks nor does a type system reduce the need to write tests if you're doing testing correctly.

[1] https://twitter.com/unclebobmartin/status/113589437016571084...

Python is a language fundamentally similar to Ruby, that has no static type checking, that's used in similar application domains, etc.... It does not have as strong a culture of unit testing as Ruby, and falls above the line (by a statistically insignificant amount) in figure 5 while Ruby is below (by a statistically significant amount).

It does not appear that the paper took automated testing into account at all. Whether greater use of automated testing in Ruby is responsible for the lower defect rate observed here relative to Python would definitely be an interesting topic for another paper.

Clojure is also dynamically typed by default and below the line. It does not have Ruby's strong unit testing culture, but it does have a design that encourages functional programming, which proponents argue reduces sources of error. It also has a culture of REPL-driven development that arguably fills some of the role of automated testing.

> Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation.

Very few of the unit tests I've written for Ruby/Python catch errors that would be caught by typical type system use in C/Java/etc. (Haskell, maybe, but...) The main thing I miss in dynamic languages vs. static is the more powerful automatic testing possible when test specs can leverage type information (e.g., QuickCheck/ScalaCheck), not an absence of need for testing.

I've only ever done property based testing in Scala, but I thought it was pretty popular in dynamic languages like clojure and erlang. What would prevent it from working in Ruby/Python?

Without type information already in the code (for parameters, etc.) there would be a lot more overhead in setting it up; it's possible, but higher marginal friction in the ideal case.

Plus, AFAIK, no one has written the libraries, making it even higher marginal friction in practice even more than than is true in the ideal case.

Type information generally isn’t used when doing property testing in Elixir, and Python has a very mature library too (hypothesis). If there’s a difference in popularity, it could just be culture. The Erlang/Elixir community is rather fanatical about reliability.

Hypothesis uses the same type annotations as Python's optional typechecker as a tool to select appropriate input data generators without additional test-specific description. That's a fairly common feature of property-based testing tools; I don't know for sure, but I wouldn't be surprised to find that hypothesis and mypy tend to be used in the same segments of the Python community.

We use property testing with Elixir (erlang VM) -- and it would translate perfectly well to Ruby or Python. I also think it's a community difference, not a technical one.

Yeah it's easy to see writing in dynamic languages with optional typing and static analysis tools (e.g. JS with TS/Flow, PHP with phpstan or psalm), the type analysis finds all kinds of different and subtle problems that unit tests probably wouldn't catch. Large integration tests often would, but are even harder to write exhaustive cases for.

Types just make everything easier to do right the first time.

That would be interesting. I also have a feeling it would be incredibly difficult to accurately control for social differences in language communities.

Yeah you'd have to take them one-by-one and see if there is some proxy if you can't measure the behaviour directly. For example, with testing you could use code-coverage or even a binary variable for the existence of tests - not perfect, but would give better results than no control at all.

Definitely. Heck, in .NET's early days, there was a distinct difference in quality of the software and of the team using VB.net vs C#, when they were 99% the same language (they had some features unique to each, but they were very few). But C# was used a lot in new projects or by people familiar with Java while VB.net was used in VB6 migrations or teams coming from VB6. That had a drastic effect on the end result, but the language had very little to do with it.

The VB shops I've seen code from tended to really like `On Error Resume Next`, too...

Sure, I've seen global try/catch blocks swallowing all errors in C#/C++ code too, but definitely not as prevalent in my anecdata.

Some developers seem to be more along the lines of "On Error Resumé Next".

It doesn't matter. They are looking at effects relative to the total effect of language choice, which they report as less than 1%. Instead of looking at these secondary effects, the main practical result is that language choice hardly matters.

JS also has more ways to shoot yourself in the foot than any other language I've worked with.

PHP? C?

Always remember to turn on warnings in PHP. I remember the joys of tracking down a mistyped variable name that PHP just instantiated and gave a null value to, which was then implicitly converted to empty string when the variable was used.

Last time I worked with those languages was in college 10 years ago so I didn't know portion of shooting myself in the foot was my fault versus the language.

I agree that JS is not substantially "worse" than some other languages.

Despite these three languages giving far too much freedom to the programmers (which some might even like), high-quality code can be written in all three of them, as long as intelligent and strict coding guidelines are thoroughly followed.

I always think of languages as a solution space, and some points in the space possessing bugs while others don't. A perfect language(impossible) would make it so you could write any correct version of a program in the language but not any incorrect version. And language constraints like static typing have the disadvantage that you're preventing a bunch of correct possible solutions while closing off incorrect solutions. Ideally closing off many more incorrect solutions than correct ones increasing your chance of landing in a correction solution.

And of course lots of out of band(non-language) processes and tools that influence this like unit testing/code reviews/strict coding guidelines. But it's always nicer when it's prevented at the language level as opposed to out of band.

Great linting tools available though. Can't say the same for many systems languages. And the static analysis tools for those tend to be quite expensive, bulky and slow.

Additionally, take a look at Fig. 8, "linus C". Those points scream "your data is bad".

Reply on news.ycombinator.com