Software Projects and Heroes: Lessons Learned from GitHub Projects
arxiv.org
arxiv.org
> A "hero" project is one where 80% or more of the contributions are made by the 20% of the developers... We identify the heroes developer communities in 1100+ open source GitHub projects.
(emphasis mine)
Of course nearly every Github project is going to have heroes--we call them "maintainers." This study is totally unsurprising and totally non-generalizable to professional teams working on proprietary software.
The use of the term "heroes" in the abstract is massively misleading and tendentious, IMO.
Edit: and maybe we can make the phrasing a bit less tendentious. Bad sign when paper titles need debaiting, but that's the trend.
We know from long experience why it's helpful to make such a change: if we don't, people will keep complaining about and bickering about the title. Editing is the only way to soothe title fever. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
This thread still comes up if I google the paper's original title, and is the #1 result if I google "software project heroes".
p.s. Welcome to HN!
By only including open source projects they've limited the ability to generalize their conclusions, which they themselves allude to in their "threats to validity" section. Changing the title simply better reflects the content of the article, which the researchers didn't do (sometimes this is due to external pressures such as university PR departments, but I doubt is the case here)
On HN, an article does not get a pass to use these tricks just because it's a paper on Arxiv. The internet game is to inflate titles to oversell the body, and the HN game is to rejig them back to scale, so there isn't a disappointing delta between what the title promises and the body delivers. We did that in two ways: by adding Github, and by replacing the "Why" bit with a simple mention of the relationship explored by the article. I'm sure it is possible to come up with something better, i.e. more accurate and neutral, and if anybody suggests better we can change it again. But if a paper is going to play the internet game—which, dismayingly, we're seeing more of, as academics increasingly have to fight for clicks in their own right—then HN is going to play the HN game. Your authors are expert in their domain and the HN community is expert in its.
I wouldn't say this change is unfair to the authors. The purpose is to steer the thread toward a substantive discussion of their work, just what any good scientist wants, and away from title nitpicks and complaints, which is boring and no one wants. I can tell you from years of experience that once these complaints start appearing, they continue unless and until the title is edited to bring down the inflammation. We can't fight it, we can only heed it—it's a force of nature. Why would that be? I think it's that people come to HN for relief from the onslaught of bait and manipulation that plagues them on most of the internet. The implicit contract here is that the front page will be a bit calmer, more neutral, more accurate. HN doesn't do that perfectly by any means, but if it's noticeably quieter and more bookish than what is typical elsewhere, that's the main thing.
You can call HN users armchair scientists if you like, but I think you'd be surprised by how many working scientists participate in this community, not to mention many who have had scientific careers before moving into industry. They are humble and don't flaunt their credentials, but when something relevant to their work comes up they mention it, and it always surprises me how many there are. It seems a good sign for the health of the community.
However, the interesting claim of the paper is that restricted numbers of developers increases code quality. However, having skimmed the paper, I'm not really convinced. I see a lot of posturing and quotations from other literature, but very little in the way of data analysis. I'd be extremely happy to be proven wrong, but I don't think this paper has done the analysis to back up its claims.
The purpose of the paper was to test and infer from a larger dataset, which open source provides more easily then proprietary.
That being said, it really seems like the research was highly guided by an agenda, and I'm not sure how much one can take from this research.
I agree that it's unsurprising but disagree that it's non-generalizable.
Companies need to break large projects into small subproject teams and one or two people on those teams need to be deeply involved in the subject matter of the subproject.
Also companies tend to treat developers as disposable entities. If you treat your engineers like crap (and don't reward them with company shares), then your company will never be able to keep any 10x engineers. Your company's products will always be mediocre, buggy, slow to develop (you will need a lot. more tests)...
It tracks with the idea that programming is like factory work.
If that 20% leave, the other 80% would contribute equally or the ration 20/80 will be preserved?
This seams a feedback loop. Once a developer has written more code than others is in a better position to write even more. If that is the case "hero developer" is a floating position, depending so much on your skill and how much time you have been working in a project and other circumstances.
> Organizationsshould reflect on better ways to find and retain more ofthese software heroes.
I had this discussion recently. If you create a team of "heroes" all become way slower than when they were in different teams.
My personal conclusion, and the way of working that I try to implement at work is that small independent projects allow for developers to be fast and reliable. If your project is correctly divided, anyone can contribute ( at their level of expertise). If you have a big complex projects then old-timers that created originally the code are the only ones that can work with it efficiently if no other developers make big changes (reducing the ability of other developers to contribute independently of their level of skill).
So, before going in search of "heroes" think if your projects are correctly divided. And think about the risk that you create when a "hero developer" is needed and leaves.
Have you found projects where work is spread thinly across many developers successful?
I am curious if there is a way to test this scientifically, large teams make some very good products.
Add in the extra work that collaboration necessarily brings to the table, and it isn't that shocking that things both slow down and get more error prone. At a certain size, though, this is worth the cost of admission.
More, it isn't like we call all projects done by fewer individuals heroes. We have authors, not "writing heroes."
And the author joke is meant to point out that the vast majority of what you read has a single person's name on it, but without the editors and other workers involved, that book would never have come about.
If true, it suggests that big companies with huge teams that try to force the entire team into equal contribution metrics are creating stress that might not help the team, nor lead to fewer bugs (big surprise).
I've never been on a team which was set up to truly recognize the value different forms of contribution, like writing tests, or removing code, or simplifying it, or other things which would not count towards the hero metric presented here. As an industry we just don't think about alternative narratives to "let's make sure everyone is an equal contributor..."
Not sure if that makes sense, I quickly typed this on my phone.
that's also because they don't want to pay the "hero" 10x than the other developers
> Much as a surgical team during surgery is led by one surgeon performing the most critical work, while directing the team to assist with less critical parts, it seems reasonable to have a "good" programmer develop critical system components while the rest of a team provides what is needed at the right time.
Other team members perform other tasks, and some of those even administrative (!), but all of them supporting the "vision" of the surgeon.
IMO this is a healthier way of communicating roles, avoiding fights and training juniors. It's better than the "everyone is replaceable" mindset companies have this day.
Have that ingrained in my mind since reading Abrash forever ago.
In all seriousness I have noticed that "business value tunnel vision" is one of the surest ways to run a company into the ground in many if not most fields and Wall Street keeps pushing it anyway to nobody's benefit. The fundamentals of the field are what you have to care about or else you will have a really bad time.
Many businesses are poorly invested and put themselves at a disadvantage.
You don't give leadership - you assign ownership, and hope leadership evolves from those responsibilities. Leadership is nurtured by helping the junior navigate issues, holding a vision for them long enough until they can realize it on their own.
There's a reason good leadership is so rare and appreciated, and it's not because it grows on trees.
Famous "Do you love your wife?"question and more.
And I don't think that having other members of the team supporting the vision of the leader is monopolizing the vision... it's just being focused on a single goal.
the surgical team is great in a crunch but long term you need to stagger your turnover to keep competency on the project high.
this isn't as relevant on team that are building projects as they were in the mmm time, today companies tend to build service software that has maybe the same shelf life but gets continuously improved instead of being dropped whole and left into maintenance mode.
Is that a problem?
I mean, the same thing happens with literal surgeons/doctors and residency, where their early years of career eventually end and they expect a mid-level position in the organization.
It sounds like a healthy industry that juniors/apprentices work under journeymen and masters at a large organization early in their career, then move up or branch out after a few years.
I'm not sure why companies expect to be able to cost-optimize away a functioning economy, but remain in business.
also while surgeon tend to specialize into one in the many fields, programmers that rise in a "surgical team" tend to become of the "full stack" variety
> not sure why companies expect to be able to cost-optimize away a functioning economy
I'm with you here, I'm not stating this is the optimal way to handle the issue, just reporting the patterns I saw on many workplaces of all kind I've ben into
Programmers? We, for some reason pride ourselves on doing something new every time around, it seems. I've argued before with my leadership that, if we really wanted something done better/faster, we shouldn't be promoting the folks that did it out of the position for the next go around. Rather, we should have them do it again. And then again. And then again. Not only would they be learning from their experiences, but benefiting much more directly.
This is similar to what Fred Brooks suggests in the "Surgical Team" chapter of "Mythical Man Month", although he's a bit harsh, I'll grant that:
> if a 200-man project has 25 managers who are the most competent and experienced programmers, fire the 175 troops and put the managers back to programming.
EDIT: We were also talking about it yesterday here at HN: https://news.ycombinator.com/item?id=19763562
Or anyone who thinks in terms like 'programmers' and 'resources'.
Put non-coders to work under the chief programmer. Let them help instead of control.
"Plan to Throw One Away"
You can't copy and paste a successful surgery from one patient to another.
> I've argued before with my leadership that, if we really wanted something done better/faster, we shouldn't be promoting the folks that did it out of the position for the next go around.
I'm confused by this - why are you (re)building the same thing multiple times?
Why are you not building on what was done before, regardless of who did it? It seems to me that either the people choosing to start over are making the choice irresponsibly, or the previous work that got someone promoted actually wasn't that great (not extensible/reusable/documented/etc). In both cases, I'd be asking management some tough questions about what they've done and/or allowed to happen.
And I'm, frankly, calling bs if you are claiming this as not something that happens regularly. Just look at all of the articles that constantly crop up of the form "why we left technology x for y".
I'm tempted to say this isn't bad, either. It is amazing what people can do retreading topics.
I think the skill differences between individual contributors are never as great as you think.
cf "Software Heroes" book - has a wireframe cowboy hat in the front cover. v good
Microsoft approach was more about having senior programmers telling juniors what and how to do their job, aka micromanagement.
Brooks is about having programmers actually coding (and documenting), and have other people handling and other important tasks such as clerical non-programming tasks, developing tools, language research, version management, doing adversarial testing, helping with documentation, etc. Some of the tasks he describe are obsolete these days, but the book is from the early 70s.
According to him, doing that "relieves programmers of clerical chores, systematizes and ensures proper performance of those oft neglected chores, and enhances the team's most valuable asset — its work-product".
Programming intent and vision is hard to communicate, so sharing code is always a compromise. It's much better to work with smaller units of code that communicate with each other using simple, clear and testable APIs.
Brooks called this "conceptual integrity":
https://en.wikipedia.org/wiki/The_Mythical_Man-Month#Concept...
I don't necessarily see it as a bad thing. It just feels the most natural, reduces friction and the need to communicate minute details. Ownership is also important. The downside of this is that some part of the project might fall apart if someone leaves, because no one else can take over.
This is the reason, i think, corporate world HATES with passion such devs. Even tho they need them to meet deadlines.
It also gives a bit too much power in negotiating to the employee which is also seen as "bad" in corpo.
Which is the right attitude, imho. Even if that person loves the company and would never leave - what if a bus runs them over?
On a large scale, consistency is more important than excellence. You need to be able to plan, "if that guy stays, the project will be done in 4 weeks, but if he leaves, it will take four years" just won't work, because you cannot rely on it. It's much better to be able to say "it will be done in 12-14 weeks" with high confidence. Sure, your whole dev team might get run over and spoil your plans, but that's really a case of force majeur.
Somewhat, yes. You want to be able to plan. Inefficiency is often the price for being able to plan. Since it's often not only about this one project, a higher price is just fine - you may have to coordinate with marketing, legal, sales, manufacturing etc, then the total price will often be so large that the 10x of the software project doesn't really matter, but if it falls flat and gets everything delayed, the damage may be huge (think global launch date missed).
I changed to a different style of programming where we break the "features" into much smaller parts called "stories" (yes, I'm one of those -- XP developer). I try to make stories about the size of a day... If you've been doing the week (or more) kind of stuff, it seems impossible to organise work at this level of granularity. In reality, it is usually possible to break things up into very small vertical slices. However, it is admittedly quite difficult.
The small slices really enables non-ownership on the team. There are a host of very nice benefits for non-ownership. It's actually a bit hard to explain, unfortunately.
First, non-ownership tends to force you into conflict early. This doesn't sounds great at first (and for some teams, it is definitely not going to work). However, it's usually better if people on the team sort out how they want to work early so that they can develop a good rapport instead of silently begrudging each other. Everybody has strengths and weaknesses and by being able to work closely together you can capitalise on that. I actually only resort to ownership if it becomes obvious that some people can't work together.
Swapping people in and out of code also results in them having to get up to speed in it. If the code is easy to understand, then people are happy and get up to speed quickly. If not... then ideally you want people to refactor the code. Umm... see the part about "conflict early" LOL. After you have a good rapport going, you'll have programmers refactoring each other's code and you'll have them be thankful for the result (honestly!). Yeah... I know.... You'd probably like what I've been smoking ;-) But it's true.
This constant vetting of code and refactoring leads to a much simpler code base. The final result is that you are able to maintain your production over long periods of time. On the best projects I've been on, we've actually improved our perceived productivity as we've added more functionality.
This seems odd, because it's the opposite of what you'd expect -- more functionality means more lines of code which means more complexity which means slower going. However, if you think of a modern OSes, languages and frameworks, we are building on huge piles of code. But the reality is that these underlying systems undergo massive churn because of the amount of use they get. In the end we get software we can build on that is stable and simple to use -- the more underlying code, the faster we can build new systems.
In my opinion, one of the most important aspects of getting to this seeming utopia is to avoid code ownership. It definitely doesn't always work and it's definitely not good for some people and some teams. But when it works, it's pretty damn amazing.
I hope that provides a bit of counter point to your experiences. I still like ownership, though... That's why I have my own private projects ;-) I'm pretty loathe to share them with anyone!
Where does this definition come from? It sounds quite typical to me. Even in large projects like the Linux kernel, if you break them down into comprehensible units and subsystems, I would expect you'd find essentially the same thing.
The vast majority of projects likely meet the "hero project" threshold defined above, whether they're good or bad. I don't know of any movement to specifically prevent having primary contributors; the main interest from the corporate world is ensuring that primary contributors can be cycled as necessary.
This doesn't seem like news in any way. I don't want to be unduly dismissive as I haven't read the paper, but I think I'm catching a distinctive whiff of "publish or perish" in the vicinity...
Short version: the OSS versus every other type of software project comparison seems very apples and oranges.
that's the gist of it, abstractions are always imperfect in some way and leave the door open for interpretation, at which point if team member don't communicate each module intent efficiently code responsibilities get mixed and entropy rise dramatically, causing more bug per line of code written.
imho an important figure on large projects that is largely ignored or conflated with "the architects" is the oracle, or the guy people ask where new feature/code should go. not in term of application structure or component design, but at a lower level, at modules and interfaces level.
I ask myself why that happens. No doubt that variations in developers' talents is a factor. But I think there are some social forces operating there as well.
I have become the hero of four projects after the previous hero had left. And I have also already arrived in projects with pre-established heroes.
Despite I've been able to be a reasonable hero whenever I ended in such position, the other times when I worked with established heroes, they were condescending over me and the rest of the team. There has always been that trend to patronize and to be a plain jerk. That was annoying.
I knew I could contribute more and maybe even to help to herorize the project. But, simply put, there weren't minimal social conditions for that to happen.
Maybe the heroing concentration would dilute a bit by improving social conditions.
You know, a similar argument could be made for history... that 80% of societal progress was created by 20% of the individuals living in that society...
"[T]he meaning of negative code is taken to be similar to the famous Apple developer team anecdote (i.e., when a change in a program source makes the number of lines of code decrease ('negative' code), while its overall quality, readability or speed improves)." - http://en.wikipedia.org/wiki/Douglas_McIlroy
"When the Lisa team was pushing to finalize their software in 1982, project managers started requiring programmers to submit weekly forms reporting on the number of lines of code they had written. Bill Atkinson thought that was silly. For the week in which he had rewritten QuickDraw's region calculation routines to be six times faster and 2000 lines shorter, he put "-2000" on the form. After a few more weeks the managers stopped asking him to fill out the form, and he gladly complied." - http://www.computerhistory.org/highlights/macpaint/
"One of my most productive days was throwing away 1000 lines of code." - Ken Thompson
"Measuring programming progress by lines of code is like measuring aircraft building progress by weight." - Pete Kirkham
My scepticism has been piqued to the extent that I call b######t.
Open source projects. Projects contributed to by people in their spare time, or perhaps with some corporate backing with developers on the payroll, and often with quite different governance models.
Should we therefore be surprised at the widely varying levels of contribution? Should we be surprised that those who do more work on the projects tend to make contributions resulting in fewer bugs, perhaps because they have a much better understanding of the projects due to their greater familiarity with them?
Whatever: OSS projects with contributions coming in from all manner of sources outside the core team aren't necessarily terribly similar to having a team of 6 - 8 working on a project full time within a company. The only similarity being that a relatively small core team do most (or all) of the work.
Colour me not even slightly surprised. My blunt conclusion is that the idea software projects "need" so-called "hero" developers is extremely questionable based on the data presented[1].
[1] Granted I've done no more than skim the paper but this comment, from page 3, amused me: "Most prior researchers deprecate heroism in software projects." Really? Do they now?
I was calling bullshit on the fact that it appeared to be drawing general conclusions about all software projects based on a survey of open source software projects, which often operate under different conditions to those in commercial organisations.
The abstract and title of the paper are somewhat baity, as others have pointed out, but the headline here now reflects the fact that the paper only represents OSS projects hosted in GitHub.
Edit: After posting this, I dug into this problem a bit more. I think code consistency is less important between modules (in a large project). But this requires a premise that these modules are independent enough and they don't need much communication each other. This line of thought is pointed out by this SO question. https://softwareengineering.stackexchange.com/questions/1906...
Big professional software development also emphasizes reliable planning, requirements engineering, and documented processes. In contrast, most open source projects begin in a cowboy style with a single person starting to code without requirements and processes and often not even much of a plan. This lightweight process implies a lot more risk and indeed most of these projects fail. That is fine. The free software world rather goes with this evolutionary approach.
I wonder if there is any company which works in a similar way. Start lots of projects with a fraction of the resources. Be fine with most projects failing. I guess not because this approach is hardly compatible with career development. Googles 20% rule might be considered a step in that direction.
I guess then the goal is to find balance between having heroes and increasing the lottery factor of the team.
Maybe this isn't what you meant, but why is it that human beings see someone they know producing a lot of value as diminishing the value produced by others, especially in technical matters?
Why? Because the mba will see a metric elevated and use that to define a norm. Now they only want to hire 10x performers. They don't know how to find them so they invent this complex hiring process that filters out so many good candidates they have no chance of ever finding that 10x developer. They settle on the fake 10x who speaks well and presents all of the right signals. The dev comes in and struggles and blames the code/past developer/way something is done and claims a rewrite is needed in a new and trendy language. This prompts a huge rewrite that takes place over a year at the end of which its determined that nothing works and that project gets cancelled. By this point that developer has been promoted to your manager.
If he and I were employed together to work on this project, a workload distribution of me writing and maintaining the entire codebase (minus one line) and him coming by every couple months to update a version number would not be a very fair or realistic division of labor.
The open source development process is very different from anything else. None of my projects at work have a long tail of contributors submitting comment typo fixes.
Once your proof-of-concept grew into something management wanted, you'd probably still be writing a lot of the code, and the other developer would be contributing, but in the realm of 20% of the codebase, at least to start (and according to this article, that wouldn't change).
Maybe the results do apply to the situation you describe, I'd be curious to know that.
That's not very precise but I think people can relate to what I said.
I can say for sure that the quality of the things that I want to do will always have better quality on average than the things that I am paid to do.
I can certainly generalize that humans are not flexible, but I don't think that's limited to programmers.
Heros are called maintainers, or creators.
This is every project. This is what this study concludes. Does the world not understand this?