The story of Google Guava and patches

Sadly, Kevin doesn't sound too good here:

And here's the last thing. Be honest: if you were going to sign yourself up for doing all that work above... wouldn't you at least want to have the pleasure of writing the code for it yourself?

Code is code. (Well, as long as it's not awful/ugly code.) I'm as happy to marshal through someone else's code as write my own.

It basically boils down to a matter of resources. When I started as an open source contributor to GWT, it was used by external developers, but not really used internally by Google, so changes made by external committers couldn't possibly break anything.

Slowly over time, more and more Google properties started using GWT, and suddenly, you had the situation where an external user could submit a patch, that passed all the GWT unit tests, but broke major Google properties (e.g. AdWords, Google Groups, Wallet/Checkout, etc) Google builds everything from head, so when you do an internal commit, not only do your unit tests run, but the unit tests from every project that depends on GWT, so you find very quickly if your patch broke real applications. This happens all the time. I commit, pre-submit queue for GWT is green (all tests past), then hundreds of other projects get their chance, and there's always 1 or 2 that break, not always because of GWT per se, sometimes because of bad code in those projects.

The problem is, there is no way for external committers to get notified of internal (potentially confidential) apps breaking on their changes. This meant that every external commit would need to be reviewed and proxied by someone on the GWT team.

Now, back when we had over 20+ people working on GWT, it wasn't hard. Now there are only 5 full time committers, and it has become a lot more difficult to keep up with the external community and support internal users.

I have been internally advocating that we "re-open source GWT". That is, we make the "source of truth" be an external repository, possibly re-hosting it on GitHub or Google Code, and fork it off from the internal version. We grant all our of best and dedicated contributors rights to administer and commit on equal footing with Google employees, and we run external continuous build systems for it.

On the innovation front, I think it's true for gwt-user, but for the compiler, I've hardly gotten any external contributions for optimizations and almost all of the improvements in speed and code size have arisen internally. This may indicate that we should run separate open source projects for the compiler/tools and the libraries, splitting them up and separately managing them.

But the GWT team IMHO was never elitist, just saddened that contributions were piling up and we lacked the bandwidth to review and commit them in a timely manner. I feel bad about it, given the time people put in, and I've been spending time recently trying to collect all outstanding patches for landing into GWT 2.5.

Patches get rejected from the kernel all the time for many of the same reasons listed in the linked post. It takes a long time for most new kernel contributors to get anything substantive in, and major changes almost never go in without huge reviews, fights, competing proposals, etc..

The only difference between that and what this post talked about is that for the kernel, a lot of the sausage making goes on in public (though far from all of it).

Dart is the perfect example of this: developed in secret, in a dark room, and then dumped on the open source community. When no one was excited about it they shrugged their shoulders, confused about what they had done wrong.

I have no interest in Java, and I really hope someone forks this project and treats it like a real open source project.

----------

The story with #guava and your patches

Guava users,

Many of you, when you request a feature for Guava, have submitted a patch to us with the implementation (or even pasted code directly into bug reports).

And we have almost never accepted any of these patches, let alone even read them. And I know this makes us look all manner of self-absorbed, arrogant and unappreciative. That's what I'd think in your shoes. So it's time I tried to explain to you more fully why it's like this.

I realize that from your perspective, you're handing us a shiny new feature on a silver platter. It should be making our decision easy, since the work is already done. It's a gift of your time and effort and you've already solved the problem and all we need to do is just accept it! Looked at that way, we're either idiots or jerks for not being interested.

But here's the part that I don't think many of you understand: the work you've done to produce that patch is actually minuscule compared to the total amount of work we have to do to put it in Guava. I know that it feels to you like you've certainly gotten us more than halfway there, but trust me, it's only scratched the surface.

- We have to work out whether the problem it's trying to solve is truly the right problem - We have to work out whether the solution presented is truly the best solution we can come up with - We have to find evidence in the internal Google codebase that users will actually use the proposed feature if we create it. If we are adding methods to our libraries that don't get used, it hurts our case when we try to argue to management that we're doing important work and need more staff. - We have to figure out how it relates to the piles of legacy code we have floating around our libraries (that you, lucky folks you are, don't even see!), and how we would deal with migrating those users if they exist. - We have to decide the best name and location for the new API. This is hard! We spend a lot of time in our API review meetings just batting names around. - We have to review the code deeply. Our code reviews are grueling and go on for many rounds. When you look at the code in Guava it tends to look "obvious", but we work very hard to achieve that quality. It's only obvious in hindsight. - In almost every case we have to completely rewrite the javadoc that first gets submitted to us. And this is very hard. Writing good documentation is probably the biggest challenge we ever face. - The tests that were first written are rarely sufficient; we're going to need to add more. When we do, some usually fail. - If the change touches on any existing functionality, we have to submit it to Google's global submit queue and analyze test results from many thousands of projects to make sure we won't break any internal users with it. - If the change goes in, we have to deal with the machinery that gets that change integrated out to you in Guava. - We then become responsible for fixing any bugs with it that come up over time, and dealing with the related feature requests it will touch off. - And the code never "stays finished' in general; we are constantly performing various maintenance tasks over our whole library (or even the whole codebase of Google), to make various cross-cutting improvements, and every bit of new code added increases that burden.

There's more I'm leaving out, but you get the idea. Guava did not get to the level of quality it has by accident, but by us being really obsessive about these things.

Now, when the patch comes from outside Google, we have additional mechanical overhead. One of us has to sponsor the patch as if it's their own, converting it into an internal patch that can merge correctly (which isn't always as trivial as it sounds), and sending it for review to another member of the team. And because we are the ones most familiar with our own style, conventions, practices and pitfalls to avoid, etc., sometimes just doing that plus "cleaning up" the code to get it ready for review is already more time-consuming than if we had written it ourselves from the start. That doesn't even mean that the code sent to us in the patch was bad. It can be very good by most standards but still need a lot of rework for our purposes.

Remember, if your feature is valuable, then we're going to want it in Guava whether you provided a patch or not. Providing the patch doesn't make it more likely that we'll decide it's a good fit for Guava -- if anything it just puts us more on guard against that seductive temptation to think "but it's already mostly done anyway, might as well!"

And here's the last thing. Be honest: if you were going to sign yourself up for doing all that work above... wouldn't you at least want to have the pleasure of writing the code for it yourself? I love writing code -- that's why I do this! -- but such a large majority of my time goes into activities like those described above. If my job were all about just applying other people's patches, I would inevitably start hating it after a while. Let me have some fun sometimes, okay? :-)

I really hope this helps to understand why your patches seem to go into a black hole. I know that no matter what I say it will probably continue to seem unappreciative and condescending, and I apologize. I do recognize that you are just trying to help. But, if you really want to help, then keep an eye out for the times when we will ask for help on a particular issue, because that's where your time and energy will really do the most good!

Rantingly yours, KB