Some performance tweaks
github.com
github.com
Not only that but the pull request is done in a fun and informal style -- a perfect example of Github's use by a Github employee =] He frankly admits that some of the changes are substantial and weren't requested or set out before hand, so there's no pressure for them to be merged into mainline if not appropriate.
It's important to note that this is an example where both sides work optimally though. I've contributed code to OSS projects backed by companies previously and it's not uncommon to end up with "dangling" pull requests -- no-one looks at it either for months or at all.
I'm still appreciative of these companies, don't get me wrong, but if it takes months for a short but critical bugfix to get through then you're not playing the OSS model properly. Either admit it's a "dump and release" or ensure your open projects are handled properly. Developers will look at you in the future and decide that's your attitude towards all your projects (see: Oracle). This ends up being a major problem when you need to win the trust of third party developers for your start-up/service/tool.
(I'm also really glad the dablooms library is getting more exposure due to this -- the initial Hacker News post fizzled out)
"Contributing to OSS is a non-zero-sum game and the sooner companies realize this, the better."
So I am asking him how could companies possibly think that contributing to OSS is a zero sum game? Its either zero sum or it is not. If they need to realize its non-zero-sum they would have to be under the impression that it was a zero-sum game.
I hope that cleared up my question. I'm sorry if my intentions were opaque, that was not my intention.
This may actually be the case in many situations, and where it's not, the possibility of outside contributions subsequently improving the technology is often overlooked.
I think I was getting hung up on how the contribution would harm the contributor and overlooking the "no return on investment" mindset.
This is a zero sum game from the perspective of the two companies providing the products.
It isn't globally zero-sum, notably the users of these companies' products are presumably better off, as are other software projects and their users who might benefit from any improvements made to open source bloom filters.
Life is too short not to have some fun in your day job and kudos to vmg for doing exactly that. For the rest of you..lighten up. Seriously.
For what it's worth, this is a phenomenon you see in most tech communities -- it happened on slashdot, proggit, and it's present here too.
In a bout of absolute irony-blindness, these Value Vacuums are far more insufferable than the very material they declared "distracting".
Yes, it's great to have the latest JavaScript ninja working on your front-end and whiz-bang Ruby folks on the back-end but eventually you're gonna run into problems that require RealHardComputerScience(TM) to fix. Or, you just throw more hardware at it and forget about it, and end up paying for that oversight over-and-over-and-over (it looks like it didn't take him much time to fix it)
Step 1. Design a test performance data set. Great if it comes from your production data! Step 2. Run your algorithm and attach a profiler. Step 3. Look at any method about 5% utilization. Can you use a better algorithm, more compact data structure, improve memory locality, or tighten the loop? Googling for fast hash functions would give you a good answer here. Step 4. Repeat steps 2-3 until you give up.
The difference with this pull request is the level of documentation that went into it. We're impressed because nobody shows these levels of documentation and humor when writing about their change. That's what's impressive.
On the other, I've seen so many devs that wouldn't even have been able to go through the process of identifying properly what to change, let alone do it, that I can recognize this pull as great.
The reality is that a majority of coders out there do not know how to properly do their job and they do not even know it (thus they don't try to improve).
(but to be fair, that's also what allow us to ask for the kind of salary we get)
I'd also second the other comments that this wasn't a great example of it: profiling and swapping hash functions is classic software engineering.
Few companies need RealHardComputerScience guys. Instead, they need people who can apply the results of CS research to messy business problems. JS rockstar ninja brogrammers can't really do that.
The problem domain often doesn't expose you to that, and that's unfortunate. There's a lot of research over the last 60 years in CS that can make our lives much easier (obviously lots of research is ongoing, too).
I know this has been discussed before, but I'm honestly mystified why this is still an issue.
The tone and humour in that post requires a large amount of confidence in the changes being made, in order to write about the humorously, but also in the author themselves to actually present their work in such a tone. vmg is perfectly entitled to do both of these; the pull request is detailed, shows clear motivation and research, and vmg seems to know his stuff. The problem is that GitHub encourages networking. The damage comes when other people who are less experienced, or frankly, less knowledgeable, copy his style and do produce noise.
I worry about a risk of imitation of this culture, but missing the crucial underlying detail and explanation that's hidden in vmg's writing. I worry reasoning with this people will be difficult because they have trained themselves to have such arrogance in their work.
I prefer a dry report not only because it is succinct, not only because it makes my life easier to understand, but also because it encourages a disciplined state of mind. If you aren't able to write about something in a mature dry tone and back it up (that is, not cover up with humour), then you should doubt your work until you can amply support it. Yes, life is short, but it's also so short that I would like to get things done; rather than have to potentially argue past people to get important points across. Lets put this creativity into making great stuff, not making great pull request comments, eh?
Finally, all of this stuff builds a record for the project. A succinct, yet detailed, pull request is much more accessible a year down the line to understand the changes in more detail. Of course, this detail should be in the commit messages (and I do criticise vmg on poor commit messages here), but every bit of writing contributes towards project documentation, at some level. The more we can create a habit to create mature, if somewhat monotonous, technical writing, I do think the better.
So no, it's not just a "I HATE HIS FUN" argument; there are more reaching concerns, no matter how exaggerated you might think they are.
"Hey, I just met you, and this is crazy, but I rewrote your bloom hashes, so merge me, maybe?"
Fun times were had by all.
It's when unsubstantiated claims are being taken as gospel does brogramming get in the way. HEROKU AND ORM ALL THE THINGS~!!!! WHAAAAAT
Edit: For comparison, here's a pull request he made to libgit2. https://github.com/libgit2/libgit2/pull/856
You have an expensive lookup. You're caching information on success/fail so that you don't have to do the expensive lookup every time. But the caches are getting large.
What you do is replace the local caches with a bloomfilter. That data structure takes a bounded amount of memory. When it says, "No, I have not seen you before," you really haven't. And when it says, "I might recognize you," it is only sometimes right. However its mistakes will not really matter because you'll do the expensive lookup.
The tradeoff is that the more data you put into a bloom filter, the higher the odds are that it will think think you might have seen things before, and therefore the less useful it becomes. But in this caching situation, it saves you work even if the false positive rate is fairly high.
"Developer profiles code; replaces slow library call A with faster library call B; ensures B does not change any important behaviour; writes self-congratulatory pull request."
Just write the facts and let them stand for themselves.
Life's too short for that shit. That's boring and dull and bland and insipid. Live a little, loosen up, we'll all have time to worry about "just the facts" when we're dead. Which, ironically enough, will probably happen far too soon, unless Ray Kurzweil turns out to be right about some of his crazy "life extension" ideas.
An an improvement though, you only need two independent hash functions to run your bloom filter[1]. Strangely enough, this isn't well known and as such isn't implemented anywhere near as often as it should be (ie. it's not implemented here).
[1] www.eecs.harvard.edu/~kirsch/pubs/bbbf/rsa.pdf
Uh, not for a while now...
Which is not to say that the general point isn't sound - MD5 was aimed at generating high quality entropy while most non-crypto hashes are aimed at generating entropy-enough fast - but don't use MD5 for crypto stuff anymore.
http://eprint.iacr.org/2012/351.pdf
It's a cryptographic MAC that's almost as fast as MurmurHash. It was designed to be used in hash tables, to protect against denial-of-service attacks from people trying to cause a lot of hash bucket collisions.
The point is that it was designed to be cryptographically sound -- and therefore more heavily optimised towards entropy over performance -- whereas the need here is for the hypothetical entropy/performance slider.
... I also read up on Murmur though, since I know SHA and MD5 already. ~3 months out CS grad myself.
MD5 is a cryptographic hash (even though it's not secure anymore for most purposes) and while it's pretty fast, you don't need any of its crypto properties, just the properties of a good quality regular hash function. Such as Murmur, or even simply FNV.
The only thing that would have made this more entertaining would be if he had submitted a similar pull request to Linus..
"hey, I changed your design and here's why XYZ" is far better than "merge me please".
Also, he got to have fun writing it, and the bitly people got to have fun reading it, so it provides a PR bonus. :-)