The empirical evidence that types affect productivity and correctness

This review is surprising in two ways: how thorough it is and how even-handed. It must have been a lot of work to put together and I'm glad you did it so the rest of us don't have to.

It's refreshing to see posts like this and http://blog.metaobject.com/2014/06/the-safyness-of-static-ty... pointing out that we don't have strong evidence on this and keeping an open mind. A trend? One can hope.

It was hard to really see the author as being objective on the subject after reading the experiment that pitted Perl/Python programs against Java/C++ programs in an application requiring "string manipulation and searching against a dictionary". I'm all for trying to prove this point empirically (though I disagree with the author's stance), but in the least you've got to eliminate as many of these variables as possible (e.g. choosing typed languages that have similar string manipulation and sane collection libraries rather than pretending like working with Java HashMaps is anything near a Python dict).

Other than that, I'd say skip the over dramatized dogma and just pick the tool that suits your problem space best.

That very first study, ANSI C vs K&R C seems like exactly the kind of test that's needed.

Unfortunately, most languages are now heavily shaped by the typechecker. That is, not many languages can flip the typechecker on or off with a switch - or have parallel typechecked and not implementations.

That was one golden chance to get people familiar with the language to try both variations.

And the effect of that? Slightly in favor of typechecked (he typed smugly). But that's not really enough. It's not clear there's even a 5% bump in productivity.

I like compile time typechecking. I like static guarantees. But there's just not enough evidence to say typechecking is a must-have. And really, I'm not sure it's possible to construct an environment to test that.

That's a good paper. The studies that involve someone writing a small program aren't too helpful, though. Typing is most useful when the type structure won't fit in the programmer's head all at once, or when there are multiple programmers and changes over time. Small scale experiments don't catch that, and are more sensitive to how well the compiler reports errors.

Looking at code on Github is promising. Several people have done that, trying to analyze changes. Of course, you only see the changes that get pushed to the server, so you're only seeing post-test results.

We seem to be converging on a consensus on typing. Within a module, automatic type inference makes life easier and programs less wordy. The newer hard-compiled languages, Go and Rust, have that. Even C++ has that now, with "auto". Program-wide type inference is hard and less helpful. Program-wide inference also means the caller's types drive the callee, which is a mismatch to the concept of libraries. So that hasn't caught on.

Expressing size remains an issue. Is size part of a type? C and C++ struggle with this and lose. Go and Rust make a distinction between arrays and slices, which seems to be working.

Type attributes are still up in the air. Read-only/immutable is common. By value or by ref is sometimes explicit and sometimes implicit. Whether something can be nil is not common, but has been tried. Rust has the "borrow" concept, which is promising but doesn't much mileage on it yet.

There's been real progress in recent years. Until recently, type systems in mainstream languages looked like either C or Pascal. Now they're getting better.

Great to come across such an even-handed review; thanks!

One quick note:

> Other than cherry picking studies to confirm an long-held position, the most common response I’ve heard to these sorts of studies is that the effect isn’t quantifiable by a controlled experiment. However, I’ve yet to hear a specific reason that doesn’t also apply to any other field that empirically measures human behavior.

In practice, I'm not actually aware of any good studies on 'tools productivity' in any other fields. (And it's not for lack of strong opinions -- grab a professional photographer sometime and ask them about Canon vs. Nikon.) I think it really is fundamentally difficult to measure things like this: getting the most out of a tool can take years, and ecosystem factors can outweigh even a large difference in 'inherent quality'.

If we care about using nice languages, it seems like the best thing we can do is try and lower the impact of these external factors. I think the situation here is actually pretty promising -- projects like LLVM take a lot of the heavy lifting out of implementing a performant compiler, and the JVM and CLR are becoming more and more hospitable to diverse language implementations. There will always be a cost to choosing an unpopular language -- but the smaller that cost is, the easier it will be for better technologies to win out in practice.

td;lr there's very weak evidence either way.

If the long list of evidence is too long for you, then it's totally worth clicking the "click here" link to jump to the author's summary at the end. The summary is readable and useful by itself -- and it is what you're looking for if you really just wanted him to summarize the research for you instead of telling you about all the relevant studies, one by one.

Most of the papers are available but first linked IEEE paper is paywalled: "A controlled experiment to assess the benefits of procedure argument type checking" Prechelt, L.; Tichy, W.F.

But it's available here:

http://page.mi.fu-berlin.de/prechelt/Biblio/tcheck_tse98.pdf

I remember Google made public some very interesting statistics about hard disks and their modes of failure, which was unprecedented in scale.

Google has lots of code in C++, Java, Python and Javascript, and their employees all have passed very rigorous assessments before hiring, so language comparisons should be less skewed by individual skill variance.

I would be surprised if they didn't already perform these sorts of analysis internally.

One big problem is programmers tend not to write programs from scratch, rather they work in existing systems.

And even though I'm a static-typing proponent, I will concede that developer productivity is likely to correlate a lot more with the state of your existing systems.

First of all, this is one of those where differences within categories are much greater than those between categories. I'd rather work at a principled Python shop than a sloppy Scala shop. If you use nulls and go overboard with inheritance, Scala becomes unmaintainable as well. I've never seen sloppy Haskell, but quality of people matters more than the language itself and that's probably largely because the Haskell community is small and elite.

That said, for me, it's about flow. (The IDE flamewar gets into the same territory. I just don't like the flow-breaker that is having to use the mouse, when coding.) Writing unit tests can be a flow-breaker. On the other hand, ripping through source code to chase down a NullPointerException (or similar error, like a "List.head []") is annoying at best and extremely disruptive when the error/failure distance is high, as it often can be in dynamic languages. I find, on average, that development is more flowful in statically-typed languages, and even if the development top speed is slower, I think that's made up by the low frequency of flow breakage.

That said, Java's version of "static typing" is abysmal and, in a typical corporate shop, you're never going to get into flow in the first place so it doesn't much matter.