Study of 49 programmers: static type system had no effect on development time

I once read an article about a technique Intel had developed for improving cooling of processors by changing the shape of the fan. I related this to some of my co-workers. One of them proceeded to tell me that this can't possibly work, backing up his argument with "reasoning" based on off-the-cuff remarks about the way air and physics "must" work. The fact that Intel had actually done this seemed to have no effect on his eagerness to continue the "debate" about this scientific fact.

I have a hard time believing that I get no benefit from using Haskell over Smalltalk, but if a body of science were to appear that cast doubt on that belief, the appropriate thing to do is change the belief, not stand around debating from imagined first principles why all the science is wrong and can't be so. Shut up, design an experiment and go prove it!

Perhaps there's little of this kind of actual science in our computer science because it would mean asking hard questions and accepting difficult truths. "The prisoner falls in love with his chains."

I'd like to see them hand a pre-written codebase of say, 10,000 lines to a bunch of students, and ask them to make large changes/new features to the codebase.

Of course, this study is fairly limited and drawing conclusions from it is probably premature. I would not base any policy decisions on it alone.

But, if static typing does not slow you down, it seems worth using even if benefits to maintainability are slight. And if they're nonexistent, using a statically typed language over a dynamically typed one still doesn't hurt you.

Besides, I think static typing is awesome. And I'm a college student--you can trust me! That should be all the validation you need :).

* The whole api consisted of 14 classes, which makes the conclusion that static typing doesn't help with API discoverability somewhat moot.

* We are not talking about Haskell, Scala or even Java/C++ flavor of static typing. The statically typed language offers no generics, no support for encapsulation, no type inference etc.

* IDE support seems to be absolutely minimal.

A methodical problem is that they seem to skip over the abysmal success rate: Only one(!) guy or gal was able to get 100%, more than halve of them was unable to implement a meaningful parser at all. To be honest, it looks a little bit like the subjects were overwhelmed by the unfamiliar syntax and/or most of them where really inexperienced (the paper says that none of them ever implemented a scanner or parser).

My feeling about this has always been that dynamically typed languages will save time on initial implementation and small projects, and statically typed languages will win on large projects and things that require more maintenance. They acknowledge this, and didn't find anything to contradict it.

Their own graphs seem to show that effect too. In some areas, dynamically and statically typed languages were equal, while in others dynamically typed languages had lower times. I don't actually see where the conclusion that a "static type system had no effect on development time" came from, it appears that it had an overall (if not very large) negative effect. Just like I would expect for a small (27 hours) project.

I didn't crunch the numbers though, maybe I'm just eyeballing it wrong. (again, I can't seem to copy-paste out of it, and I don't feel like reentering all of them)

I just started diving into Ruby. I'm about a month into it, after having spent over a decade doing Java and other static languages. The productivity boost for new development in Ruby is very real; I converted a 10K LoC java program to < 1K LoC Ruby in about a week, despite not knowing the language well at all. It almost feels like a "runner's high" to be so incredibly productive. But there are several things that I find frustrating about dynamic typing.

In dynamic languages it's very hard to navigate existing code and libraries. In Java, you can click on a variable and instantly navigate to its declaration and class definition. In a well-configured environment (say Intellij + Maven) you can even click on imported libraries and drill right down into their source, even the JRE. It's totally natural to know exactly what a method takes as arguments, and what it returns.

In Ruby it always feels like a mystery. When you ask the IDE to jump into a library, it pops up a dialog of 20 choices and makes you guess which one to open. It feels like the old days of running find/grep over the source tree. And if you somehow manage to guess correctly and find the code you're looking for, you often have to read the comments (if they exist) to figure out what it expects/returns. I'm sure a lot of this is due to my inexperience, but code navigation is really painful in a dynamic environment. I can only imagine how hard it would be in a non-trivial project after a couple of years with a few different devs.

Of course refactoring is way better in a static environment. Even if you throw out the IDE, you can more or less instantly find where a piece of code is used by simply renaming it and recompiling to see what breaks. In Ruby you have to rely on tests for that kind of thing, instead of getting it for free from the compiler.

Don't get me wrong- I'm excited about Ruby, and I think dynamic languages will play a major role in the rest of my career. I just wish that code navigation could be better.

From my personal experience, dynamic typing usually works well with 1-2 developers and continues to be useful until about 4-5 developers (static typing too for that matter). After that, static typing usually wins pretty consistently.

My personal preference is static typing. I find it makes the code more readable and catches a lot of oopses early, at compile time. Otherwise you just end up writing a ton of test cases and hope to catch the same issues at runtime. So you have less protection (you have to catch it with a test), and you just end up spending more time writing tests anyways.

I do understand the benefits of both, I just prefer static typing because of the way I work. Both work and have their places. Like everything else, you need to use the right tool for the right job AND the right people.

The main disadvantage of dynamic typing for me is the lack of verifiable documentation about what data a function needs, and which data comes out of it. This becomes a problem when

a) the data is complex (e.g. dictionary of lists of items with certain properties)

b) I haven't looked at the function for a while.

In those cases, I find statically typed code easier to reason about.

On the other hand, it absolutely maddens me that static type systems force me to spell everything out, even where it's trivial. It slows me down, it bloats the code (especially when you have to create a new class for everything) - which again makes it harder to see the purpose of the code instantly.

That's why I decided to make my own little pragmatic experiment: In cases where complexity is expected (or experienced) - and only there- I use the Clojure pre- and post-conditions to check the shape of selected parameters and/or its return value. It looks like this:

(defn german-holidays

  "Returns map of german holidays, in the year of d."

  [d]

    {:post [(like {(date 1 1 2012) :easter} %)]}

   ...)

  [c start-date end-date]

  {:pre [(like {:appointments {} :new (list)} c)]

   :post [(like {(today) {:occupations [] :appointments 

   ...)

I'm not yet sure how this experiment will fare in the future, so any suggestions or warnings are appreciated.

1) a REPL. Just give it a whirl and see what you get.

2) intuitive method names and good abstractions. If I call @user.posts in a Rails app, I can assume I will get an Array-like object that I can treat as an array without any problems. The actual type does not matter 90% of the time.

For the benefits of static typing I would rather look into something like Haskell which has a much stronger notion of type and builds many language features around this notion. It's probably harder to design a reasonable study comparing that with for example Smalltalk, though.

It seems the only question this experiment could possibly claim to answer is whether or not the additional finger typing required by static languages affects the coding speed of these particular 49 undergrads.

That programmer quality shows way stronger effects than the measured effect of a typing system would suggest to me that

- the measurement was flawed?

- programmer ability beats language choice by orders of magnitude?

Or do I make a category error here?

You take Smalltalk for example, as it is the original archetype of a language that benefit from blurring the lines of the type system. A language like Haskell on the other hand was design with the intention that the programmer would use the type system to do a lot of heavy lifting.

Of course, no real-world language is "pure" in that sense. We've graft type systems onto Smalltalk's "way of doing things", and the GHC Haskell compiler now allows you to treat type errors as warnings.

They take 49 students and train them in the language - 16 hours for the dynamic version, 18 for the static, because of "the additional effort for teaching the static type system". The students have 27 hours to complete two related tasks. All activity was logged, so they could look at the time from a test run failing with type errors to a successful test run.

- Scanner task: overall, students using the dynamically typed language were significantly quicker to complete a scanner that passed the tests - on average, about a third faster. Within that, the time spent debugging type errors doesn't differ between the groups.

- Parser task (builds on the scanner): The implementations are tested after coding has finished, showing no significant difference in success between the languages. In both groups, about half the students didn't manage a meaningful parser at all. In this part, students with dynamic typing spent significantly less time debugging type errors.

Here are a few things to keep in mind, IMO:

- Good/capable/flexible static languages are much much harder to design than dynamic ones. Case in point is Haskell, where hundreds (thousands?) of researches have been working for years to widen the area of expression vs. area of safety. The devised language for this study seems very minimalistic to be useful. Most dynamic languages, on the other hand, pretty much let you do anything even when they have a simplistic design.

- I find that people take longer to become fast/useful in static languages. While you can just throw things together and tweak until it works with a dynamic language, a static one will force you more to know what you're doing. I had a semi-technical colleague who had a much much easier time getting up to speed with Python than with any other static language. Therefore, unless the study involves a long prep time, it will naturally be skewed towards lower devel time in dynamic.

- Static languages start shining as the project size/time-line grows. Key affected areas are refactoring, intra-team communication and incremental improvements on the code. To test for this, it would be interesting to stage a time lapse where developers are asked to modify their previous parsers after, say, a month of total separation.

I think a more interesting study would be to hire 20 highly decorated Python devs and 20 highly decorated Haskell devs and give them a few different computational tasks, overall adding up to a large # of hours. Naturally, the study needs to limit usage of 3-rd party libs and take other normalizing measures. Then we may get a sense for what the state-of-the-art in both fronts have to offer.

I do like static type systems (like Scala) more than dynamic ones (like Ruby), and always thought it would perhaps people slow down. Good to hear it doesn't.

One would need to look into brown field development, if there is any positive/negative effects (I assume positive ones)

To quote:

We measured two different points in the development: first, the development time until a minimal scanner has been implemented, and second the quality of the resulting software measured by the number of successful test cases fulfilled by the parser. In none of these measured points the use of a static type system turned out to have a successful impact. In the first case, the use of a statically typed programming language had a significant negative impact, in the latter one, no significant difference could be measured.

If static typing wasn't useful, I don't think anyone would bother with types. At the same time some static type systems are restrictive while coding, so you're seeing more and more type inference in languages.

Go does it by allowing one to declare a variable with an expression " a := 1".

C++ 2011 has it with "auto"

Haskell is pretty darned good at it, and a lot of people don't write the types of their functions, though I think it's good practice to make sure you and the compiler agree on what you've written.

That said there's languages like Clojure which are dynamically typed in a sense, and I believe pay a slight performance penalty because they have to use reflection to behave that way. There's ways to annotate types in Clojure to get around that. This is an interesting case because Lisp is typically a dynamically typed language.

At the same time there's awesome languages and environments like Racket that have statically typed and dynamically typed versions of their scheme dialects.

My feeling is that a ruling on whether types are useful or not is sort of a pointless discussion. It's not even true that "the jury is out". You're just choosing tools with different properties and some make sense for some situations and others for others.

With dynamic typing it's much more difficult to create large, reliable codebases. There's too much implicitness among the interactions of different actors, modules, and programmers.

With static typing it's much more difficult to create any codebase. The exploratory nature of programming means that you will have to constantly change and redefine the static types, signatures and interfaces.

What you want is a language where you can start prototyping dynamically and later on when the APIs stabilize gradually you can then stamp static type checking on top of them for cementation without actually having to rewrite anything, possibly in another language. When you can selectively force a freeze on a certain piece of code or API then you can gradually move from dynamic to "static enough" typing as the part of your development process.

The programmer's expertise is an important one and it is difficult to take in account. Static typing is to check code validity. If the programmers make no mistake, then of course specifying types is an overhead. When it comes to collaborative work, strong typing can save a lot of debugging time, meeting time and documentation writing and reading time.

Another aspect is the resulting code efficiency. Code is programmed once and executed many times. It is much easier for a compiler to generate efficient code when it is given more information to do so by the programmer. This benefit of the typing overhead is not taken in consideration.

As repeatedly said, use the appropriate tool for your application.

Or, y'know, according to any basic understanding of science whatsoever. Jesus.

They're essentially trying to test whether static typing optimises development time, and as with any optimisation, you only see a substantial increase if you're optimising the bottleneck. Static typing won't show any speed benefits if your coders are being slowed down by weird abstractions, poor tooling, or analysis of the problem.

The experiment didn't focus on use of the type system, but nor did it actually test anything close to a real-world scenario. All in all, it seems worthless to me.

The mean development time for dynamically typed languages is about 30% shorter in his first experiment, and it is about 20% shorter in the second experiment.

Statistical significance test determines how likely this difference would be if the difference development time in a large population was 0... It doesn't mean that the true effect actually is 0... it's just addressing whether we can completely rule that out.

A more accurate summary of his results would be "Dynamic type systems allowed 20% to 30% faster development in my experiments, but I had too few participants to reject equality."

Try comparing the time one needs to change the codebase written in ML and the one written in Lisp and come back. It is experimentation and change that is slow in static typing.

> [Decreased development time means] that the quality must be improved since developers have more time left in their projects.

What kind of nonsense is this? What's this assumption based on? Hearsay? Shorter development time = better quality? What?

> [Increased development time means] decreases the code quality.

Again: what?

Caution: Not an academic research paper, but a real case study from Amazon.

[1] https://sites.google.com/site/steveyegge2/is-weak-typing-str...

The fact that type system had very small cost for initial development of small system should be seen as an invitation to use strong types.

Its the bug fixing time that it affects.. A dynamic language is harder to bug fix than a static one..

Of course, the static typing proponents you find on HN are generally not advocating incredibly limited type systems like java or the one used in this "study". I find it hard to believe that a language with such a limited type system is representative of languages with useful type systems like ocaml or haskell.

Hopefully for the next experiment they have them tackle a longer project, and have the programmers tackle short problems in multiple languages to 'rank' them.

2. They ought to check on maintenance effort and time with different type systems as well.