Fact and folklore in software engineering
morendil.github.com
morendil.github.com
It's nominally true because many programmers have zero or lower productivity, sometimes due to no fault of their own. If you work on a project for three years, and then it gets canceled, you could say your work had no economic value because it never got in front of customers. If some PHP hack manages to pound something out in a month that brings in $100,000 of business, he looks like a hero, no matter how bad the code is.
In my view, superproductivity is about alignment with your environment. If you work for 12 months a year, but spend 3 of them on side projects that go nowhere, spend 3 months being a sysadmin, and then waste another 5 months on reworking a GUI, there goes about 90% of your productivity.
The "Mythical Man Month" says that about 20% of the time on a software project is spend coding, the rest is requirements work, testing and that kind of stuff. A superprogrammer with supertools might be able to do that in time that approaches zero asymptotically, but unless you can get rid of the other 80%, project cost and time don't get much worse, even in comparison with a coder who is half as productive as average.
The moral? If you want to look like a genius, find some place where (i) requirement work takes little time (and wastes little time downstream with changes) and (ii) testing, deployment and all that is minimal.
Example: I had some large csv files to import into a rails app. Instead of writing my own importer and running it with script/runner, I scripted a quick importer ontop of mysqlimport. Mysqlimport ran about 1000x faster than a custom solution.
I guess this averages out. Just that some people seem to have incredible mental stamina.
You don't win a race by "driving fast", you win a race by never driving slowly. It doesn't matter how fast your car is if you spend 30% of your time stacked up at intersections.
Similarly, programmers spent a lot of time dealing with various barriers to productivity. (For instance, bugs you can't fix) If you want to increase your productivity, you need to attack these barriers by every means necessary.
Of course, there's some real variation in the averages that people will settle down to over the course of hundreds of debugging tasks, but an individual's productivity on debugging tasks varies so much that I'd be hard pressed to say much at all about the individuals until I'd seen them do at least a few trials.
It may be the case that the differences in the average times people take do end up being on the order of 10x or more, but it would take a lot of observation to say that for sure (exactly how much depends on what sort of distributions we see when we measure this stuff).
Interestingly there are articles out there which suggest that one distribution (of competence, rather than productivity, but intuitively you'd expect the two to be related) is actually bimodal rather than normal, "the camel has two humps":
http://www.codinghorror.com/blog/2006/07/separating-programm...
Since when did testing and deploying become "unproductive work"? If getting reliable, tested code in front of customers is not your job, what is?
Old school people might call it getting stuck in a rut. It's much easier for me to get stuck in a rut doing testing and documentation, compared to doing something new and exciting.
I'm curious as to whether developing automated testing and deployment stuff would count as doing testing and deploying, or as developing exciting new stuff.
And for me, developing the automated stuff is still not particularly interesting. I guess if I were designing cars, my interests (and thus most productive department) would lie with building the car itself (engine, body, drivetrain, etc.) rather than the safety system or QA. Maybe it's just my personality.
Personally, I think the author is unintentionally also pointing out the problem of relying on studies of programmer productivity: it's next to impossible to measure. I seriously doubt that there is no valid research on this because no one has gotten around to it. Too many peoples' businesses rely on understanding programmer productivity inside and out for that to be the case.
Personally, I agree with whoever it was that argued that we need to view this as a soft science like psychology or sociology. We need to focus on working in spite of our imperfect means of measuring productivity rather than dismissing all of our research based on it. Such is the nature of studying the human mind.
PS: I have completed the same task as other programmers in less than 10% of the time. However, assessing overall productivity requires that someone is consistently faster on a wide range of tasks not just that they happen to solve something in 10% or even 1% of the time.
I've never taught a CS course, but back in my college days I helped a friend with his CS homework by taking advantage of weak file permissions to copy other people's homework. The shocking thing was that, looking at a small sample, most of the programs didn't work correctly. To complete the assignment our way, we had to fix bugs in programs that other people wrote, which is fantastic preparation for a professional programmer ;-)
I don't think the concept of productivity can span multiple fields. Does a C programmer have the same productivity as a Ruby programmer? On the other hand, maybe some part of a Ruby project needs optimizing, so a routine is written in C. Maybe there can be a definition of productivity that crosses languages... but we don't have one yet.
Now back to Terminal and vi so I can be productive again...
Part of my point was, though, that nobody should need to do an objective study. :) Because it's pretty frickin obvious to anybody with a brain who works in the field, in my judgment. And it's not limited to our field. It's a human-wide phenomenon.
So I would choose to politely disagree. You can measure productivity.
Am I negatively productive under your measures?
1. Familiarity with the library set / language
2. Familiarity with the problem space
3. Familiarity and speed with tools/programming environment
4. Research speed (and research tools/materials)
5. Mental Program structural planning speed
6. Previous Experience with the specific problem
7. Knowledge with time saving programming structures
8. Skill with mathematics and algorithms
9. Computer Hardware & Internet Connections
10. Ability to choose the best tools for the job
11. Learning speed
12. Motivation, energy, ability to concentrate for long periods of time
And that's listing mostly external factors that are somewhat obvious, not some less obvious speed of mental thinking or something similar.
You can define developer productivity as speed to complete an identical project compared to another and the quality/maintainability/readability of the software they produce to match the project requirements.
Anyways, it's really an unfalsifiable claim, isn't it? For any study that doesn't show a 10x difference, you could claim that it didn't have any truly "good" programmers.
The burden of proof should be on those who make the claim.
The bottom line? Studies using a group of undergraduates on a day-long project are worthless.
The other thing I have seen are efforts to measure professional programmers via time sheets, even had to do it myself, usually in the guise of getting us to produce more reliable estimates. Not more accurate estimates, but more reliable in the sense that Jack is always off by 4x while Jill is consistently off by 1.5x. These efforts usually died because programmers resisted the hassle.
Other measurements have been equally worthless. Measure by lines of code? You get lots of blank line and comments. I have even worked on projects that required 2 pages of comment for 5 lines of code.
indeed! computer science is a social science! http://www.achangeiscoming.net/docs/cssocsci.html
"It's important to note that (as far as I can tell) the author doesn't actually debunk the existence of precognition, only the claim that studies show precognition to exist."
Yes, but quite vacuous. The author can't prove the non-existence of evidence supporting the 10x claim. He was critiquing our folklor-ish belief in this 10x claim, without evidence, and our gross indifference to this fact.
Let my try another tack, proposing: excluding some bottom <10% that are incompetent, and those with less than 5 years of experience, all programmers have exactly the same 1x productivity capability; all apparent levels of difference are really either ramp-up on new technologies or flaws in development processes and tools. <citation of irrelevant study on something else> <citation of myself, citing the aformentioned study>.
... do you now believe in this 1x claim? No? Hmm, perhaps I need to get it quoted more and then you'll believe it?
The author's point is that "quoting it more" and "it seems reasonable" are the only basis for our belief in the 10x claim. But neither are scientifically, empirically relevant. We should follow up with further studies to confirm the effect, and then investigate the cause. But we don't. And because we have not, we don't have any scientific basis for believing this 10x claim.
This isn't a matter of "dismissing all of our research based on it" -- there is nothing to dismiss. There has been no sufficient research.
Oddly, my initial misread is inaccurate. There are far more studies supporting the existance of precognition than supporting this 10x claim. ;-)
Even programming at my very basic ability level (actually I never considered myself to be a professional programmer * ) I've seen a number of people who simply never get anything done, and are 10x less productive than I am. The sort of people who are the heroes of thedailywtf.
They never do any better than tinkering around; with some luck they may actually implement some simple code if given very precise requirements because they miss the basic impetus, or the necessary comprehension to start by themselves. Maybe in some huge java drones team, they may add some value, I don't know for sure, I've never worked in a team of more than five programmers anyway.
OTOH I've worked with some guys, thumping the keyboard 16 hours a day for months straight, getting working production code out of the frigging door every single day. So I dare saying I've seen it all : the 10x programmer, the 1x programmer (myself and most others), and the 0.1x programmer (I don't know how many).
All of this sure isn't scientific fact, but simply what I witnessed myself in the past 25 years. It's enough information to guide me through :)
* I'm a professional dilettante :)
I would love to hear an example of that as that seems a pretty extreme claim.
In the cases I've seen what makes people a lot more productive is experience. At Google, Ken and Russ ran circles around me in terms of writing compilers, but I had only written half a dozen toy compilers while someone like Ken has written many more production ones.
My current co-worker Mikko just spent a few weeks re-writing the code to extract a mesh from our level set data. It's a distributed algorithm and our literature search didn't indicate it had ever been published. Starting with someone without domain experience it would be several weeks until the algorithm made sense. It probably would take years to replicate the feat.
Programming is a very specialized endeavor. Some parts of more than others. My own experience indicates that individuals can have magnitudes of difference in their performance when applied to some of these specialized areas.
It's certainly not that you're 10x better, it was homework. Your tutors expected you to be able to complete it as they'd spelled it out to you in the lectures.
We're talking about one programmer being 10x better than their peer, not of one generation being more scientifically advanced than the previous.
You will note that the expert is not quicker by virtue of pulling in more sophisticated tools or large libraries. Rather the expert is faster due to knowing how to solve the actual problem.
Another one: http://www.paulgraham.com/spam.html
For many years hundreds of programmers worked on anti-SPAM systems that weren't as effective as the very simple method PG proposed.
If anyone else can think of more I'd love to know them.
The very fact there are articles explaining previous solutions to these problem domains is why I just don't believe anyone would ever take a year to get up to speed with something an expert could do in one day.
I can make a great spell checker or a great anti-spam program that takes an expert a day to program in less than a year by just googling it.
There are so few programming domains that aren't fairly transparent where a domain expert has such a massive advantage that a domain novice couldn't gather sufficient knowledge in a year to rival the other programmer's one day effort, given the same level of intelligence, etc.
Moving back to programming, debugging specifically, I've had people spend weeks stuck on a bug, only for another programmer to find it in hours or days.
Citation itself often presents as a modality,
with an associated degree of confidence.
(I realize this is only DH2, but it's such an extreme case that I can't help thinking how lucky we are not to have programming ordinarily discussed in this way.)
"Citation" is the topic being discussed. It's a technical term that refers to how scientific papers refer to other scientific papers.
"Modality" is the concept I've been introducing for three paragraphs. You're supposed to know what it means by then, or I've not been clear enough.
"Present as" is an intransitive verb phrase. I could say the same thing, perhaps more simply, by saying "you can think of a citation as a modality". Perhaps it's this phrase that has thrown you?
[EDIT: I've changed the article to try and make the sentence clearer. Thanks for your feedback.]
It seems to me that "degree of confidence" is clear enough, and that it's clear enough that a modality can have a degree of confidence associated with it.
So the article introduces two technical terms you're perhaps not familiar with, but I'd suppose someone who can read a CS paper or write a program can cope with that much.
I hate to break it to you, but people (especially on a site like HN) have a tendency to skim over text. I hate to repeat myself as much as the next guy, but when you're communicating with human beings it's quite often that you have to beat a dead horse.
The other thing is that the overall sentence isn't necessarily very difficult to read, but the fact that there are so many $10 words gives the impression that it's more difficult to read than it is. So most people (like me) don't try.
With the above noted, I don't know if there's any way it can be improved. Perhaps that's the best way to say it. But those are the reasons others might find it difficult to read.
A better starting point for programmers and computer scientists would have been modal logic, which uses the modal operators of necessity and possibility.
For example, classical logic uses propositions. I can say "P" in classical logic. In modal logic, I can say "P", "Necessarily P", and "Possibly P", where logical necessity and possibility are modalities. See http://en.wikipedia.org/wiki/Modal_logic -- it's a pretty good overview and it links the logical and epistemological senses of modality.
That is the kind of language which made many European intellectuals in the later half of the 20th century notorious. The problem is not just an unusual vocabulary and elaborate sentence structure, but fundamental incoherence and haphazard use of undefined terms. I don't believe the linked article under discussion is really in the same category.
I would try to avoid words such as "modality"---"qualification" might be better---but the point he makes is clear. He spends several paragraphs defining what he's talking about and then makes a case. It actually follows the lemma, lemma, theorem formula closely. He goes on to offer evidence for his point. Summarized, he says: over time citations can lend credibility to an otherwise unproved statement. He then says programmer productivity disparities haven't been conclusively demonstrated. That might have been a better starting point for the article.
He's guilty of disorganized writing, and being influenced by literary theory, but his point is otherwise legitimate and capable of being proved or disproved. (I don't know how much he's edited the article since.)
I love the self-awareness implied by referencing your own disagreement hierarchy, and humbly wonder whether the following insightful remark may be pertinent here:
“Someone who has a chip on their shoulder about some topic might be offended by a tone that to other readers seemed neutral.”
I expect that this 10X figure is also applicable to things besides software engineering. I'd guess that auto mechanics with 30 yrs of experience running their own auto shop are 10X more productive at solving problems than new guy at the dealership. (Note: solving problems, not just replacing the part. Knowing what part to replace) The Ph.D. who's been researching for 30 years probably is a lot faster at finding solutions than the guy who just passed his qualifiers.
The focus was more on the language factor, and some programmers were students while others were pros, and in some they were self-selected; but still, >10x variation in development time among the Java programmers. 30x if you allow all the above differences. The 10x thing gets quoted way more confidently than the amount of study justifies, but it's not just made up.
The problem with reading off a 30x variation (or even >10x) from Norvig's results is that it discounts the possibility that the same person might take different amounts of time for tasks (relative to the others in the group).
For instance, you might have people in the group that have already worked some variation of that problem. Or you might have some that had colds, were distracted, or just do badly on that sort of enumeration problem, but are killer at other problems. Every one of these possible discrepancies cuts away at the expected magnitude of the actual overall productivity difference between programmers, and should be accounted for.
Of course, since that difference is not what was being studied, they didn't set up the experiments to gather the data we'd need to best estimate the real productivity differences, so it's hard to say what we should conclude...I'm sure there are people that are 1/30th as productive as the best programmers, but what we'd really want to know is how typical they are, and what the overall distribution looks like. That's far less clear.
First you have to define what things like "productivity" are for programmers, and to my knowledge no-one has really done a robust job of even that yet.
Then you have to establish who the "best" and "worst" cases are. After all, I suspect many of us would agree that some programmers make a net negative contribution to their projects, taking more time away from more efficient programmers to fix the resulting problems than it would have taken the stronger programmers to do the work themselves in the first place. This may be inevitable, given the learning curve involved in a field as complicated and diverse as programming.
That all said, I will just mention that Glass also covers this subject, with various other citations, in "Facts and Fallacies of Software Engineering". Given that his final fact is "Many researchers advocate rather than investigate", a criticism very much along the lines of this article, it would be interesting to know whether his sources stack up to more robust criticism if anyone has access to them.
It is more like "social engineering" or "financial engineering". More to do with ingenuity, than engineering as a practice.
Engineering is an application of scientific - mathematical and physical knowledge to create something. If you don't do the Math, you probably are not an engineer. There is little room for subjectivity.
So, I guess most programmers (if not all) are software technicians. They have practical knowledge, but not the proven theory behind it.
So you will have "Fact and folklore in software engineering", until you can synthesise "something-driven development" using knowledge mathematics and science.
Currently you have some ideologies and studies on them.
In control engineering for contrast to solve a problem, for example a controller regulating the flow of fuel in an engine you have the following procedure:
1. You make a model of the open and closed loop system. You know how do the processes look like in derivatives and you can make transfer functions and state-space representations.
2. You correct the system according to specification, adding to the models.
3. Knowing the physical properties of the different parts of the system you are left with some choices of implementation, but they have different trade-off (physical and economic). You talk with the client and make the appropriate choice. :-)
So, you have the heuristics to solve the problems all build on both practice and science and you don't have the engineers split in schools or ideologies :-)
I guess it should be the same with programmers.
Also, it should be kept in mind that there is a chance that the original study was contaminated by bad programmers. The truly bad - those who don't really know how to write code at all - are going to vastly underperform those of average competence.
If you had a test that screens out basic competency (an ability to handle CS 101 tasks like assignment, for instance) - how wide would the range be?
Certainly based on 1 study from the 60's... we can't make a good judgment. It's not even a coding study! Even if it was a coding study, it doesn't measure how well they work in teams, or how well they adapt to new languages, or the myriad other things that matter in a programmer's career. . Here's a question that deserves study - what interview techniques can make productive teams? What productivity difference is there between those good at brain teasers (programming ones or otherwise), and those that are bad at them? What about those given syntax questions? Those who feel like a good cultural fit? General IQ?
And then he tears apart the study with strange assumptions. "Well they were debugging, not programming." Because no programmers in the real world spend a significant amount of their time debugging. I'd consider that to be a significant amount of time for a programmers day. Wish I had code complete in front of me, because I know it did do a break-down of developers time. The one that said that most developers spend more time walking than reading software books.
If you're going to refute McConnell or even Peopleware, you've got to do a study. You can't just say "well cubes seem more productive than an office."
> If you're going to refute McConnell or even Peopleware, you've got to do a study.
What McConnell has published isn't empirical, or scientific; argument suffices to refute it.
That goes for DeMarco and Lister, too - Peopleware was a popular book, not an empirical study aiming at establishing scientific fact.
Your debugging criticism is somewhat valid, but the context here in that the original Sackman paper was mostly testing the difference between "online" and "offline" programming (i.e. programming whilst sitting at a computer or not — nothing to do with the internet!): see, for example, http://dustyvolumes.com/archives/497 and http://dustyvolumes.com/archives/500
The research was on whether debugging whilst sitting at a computer would make you more lazy, which is an interesting experiment for its time, but isn't really that connected to the modern concepts of debugging vs programming.
A far better refutation would be be: Well I ran study Z, and it shows...
(Not that I think the article's claims are as outlandish ad Global Warming Deniers, or that developer productivity has been studied nearly as much as global warming.)
To the contrary, a very important part of science is a lot of lazy onlookers scouring what others have done and shouting "You didn't prove what you think you proved!"
Granted, it's always better when they are able to do the experiment right, but failing that, knowing that the original one was done wrong is still extremely valuable information, especially when it's a result so widely quoted.
"Well they were debugging, not programming." Because no programmers in the real world spend a significant amount of their time debugging. I'd consider that to be a significant amount of time for a programmers day.
To me, the problem with taking a study on a single round of debugging as meaningful is that debugging, overall, is a higher variance activity than writing fresh code.
Say there's a large group of exactly identical programmers, who would each take an average of 50 minutes to find a bug, but the amount of time for any individual to find a particular bug ranges uniformly from 10 to 90 minutes. If you give each of them a single bug to find, you're almost definitely going to measure "productivity ratios" that come close to the limit of 9x (with a big enough group). But all of these programmers are identical, so we damn well better not be publishing that as a result that "Programmer productivity varies at least as much as 9 to 1". We need to run more tests, or at least otherwise parse the data in some way in order to figure out what we can really conclude.
In comparison, the act of writing fresh code will usually have lower variance, because it's less of a "search through unfamiliar shit" task, so we could probably take inferences based on single trials there a tiny bit more seriously, though it still doesn't justify sweeping this issue under the rug.
I haven't read the study that we're referring to, so maybe I'm wrong, but since I know that this claim wasn't the primary focus of the paper (they apparently just noticed high internal variation within each of the two groups that they were comparing), I have a sneaking suspicion that we are talking about a single debugging task, and directly comparing the best and worst performers on that task.
FWIW, Norvig's results (http://norvig.com/java-lisp.html) are more meaningful as far as drawing distinctions between programming language productivity (though they don't distinguish whether the languages used are a cause or an effect of productivity differences), but they also cannot be directly used to prove that, for instance, some programmers are 30x more productive than others, because the variance within an individual's productivity is not accounted for.
And yet, his point is made. :-)
Doing that might involve getting clear on the terms. "Productivity" is too abstract and ambiguous to be used as-is, so we first need to say "here is what we are going to measure".
With that said, this is not trivial even for activities where the metrics seem a lot clearer, e.g., basketball or basetball.
The central message that I remember from Peopleware is that software teams are more about people than about numbers.
Performance is going to vary, often wildly (often for the same people at different times). I loved the anecdote of the woman that wasn't very productive herself, but was a "catalyst", maybe simply a fancy way to say she was fun and nice to work with.
Unless you are Google and have the right process to detect the best and resources to attract them, your best bet is to treat people well and apply common sense... as opposed to apply hard metrics and put pressure, demoralizing your people.
For caned results that you could look at I would think ICFP programming contest or similar would have numbers though a quick look hasn't turned anything up.