Twitter open sources its MySQL secret sauce | Hacker News Reader

Twitter open sources its MySQL secret sauce

gigaom.com

147 pointsnoinput19 comments

Better link: http://engineering.twitter.com/2012/04/mysql-at-twitter.html

Did I read 'fork'? If you want the open-source cred, make an attempt to get your patches upstream. Google has done it for MySQL (pre-Oracle).

Google has their own branch: https://code.google.com/p/google-mysql/ Facebook too: https://code.launchpad.net/mysqlatfacebook

I don't know if it applies in this situation, but certain cases make sense to fork instead of patch. If they are including features that are orthogonal to general MySQL feature set, it shouldn't be patched.

Considering their size and unique requirements induced by that size, I imagine a lot of these changes do not apply to small MySQL installations running WordPress (and in fact may be highly detrimental).

I think all of the changes pretty much do apply generally, but for us it's easier to go ahead and publish publically to be able to pass these changes upstream, rather than pass them off privately and wait potentially a year or more to get them back in a public release.

We want to be able to collaborate with Percona, Facebook, Google, MariaDB, etc., on some of the actual changes, to make things better for everyone. This is the first step towards that goal.

If trying to get their patches upstream doesn't make sense to them internally, it probably won't make sense to others.

This is actually not really about whether it makes sense, but about timelines. We expect some of this to eventually make it to upstream MySQL, but likely in a version newer than we need it in (5.6 instead of 5.5) and not for something like a year.

It is a fork in that it is a parallel line of development but from reading Twitter's blog the desire is to get some or all of their work upstream. Releasing the fork like this gets them some early OSS & engineering cred while being able to start the conversation on their modifications.

We look forward sharing our work with upstream and other downstream MySQL vendors, with a goal to improve the MySQL community.

http://engineering.twitter.com/2012/04/mysql-at-twitter.html

It's a fork in the sense that we're developing without an immediate requirement to get MySQL/Oracle to accept our changes. We hope they will accept the changes, of course, but it isn't and can't be a requirement for our own progress.

This might be ideal, but it can be very difficult to work with an upstream company like Oracle (who are the copyright holders to MySQL).

Maybe we'll see another LibreOffice?

Link to github repo: https://github.com/twitter/mysql

Is there any overlap between the work done in Gizzard and the other popular MySQL forks (Percona, MariaDB, Drizzle) or are each of these solving different (incompatible) problems?

Gizzard is sharding framework that sits on top of, as a separate layer from, MySQL (or any other storage backend, like Redis). It's open sourced right now at https://github.com/twitter/gizzard. Twitter's MySQL fork for now is just a way to get in bug fixes and extra features that are needed but not available upstream.

Yup! And our changes are quite complementary to Gizzard, since it's one of the larger use cases at Twitter for MySQL. More to come in the future to better support big Gizzard systems!

Thanks for the clarifications. And the work :-)

Is this something that lives in the same space as http://code.google.com/p/vitess/ ?

Not quite - vitess is a proxy that sits in front of MySQL and adds additional features, whereas Twitter are releasing their own fork of MySQL which contains optimisations they have developed for their environment.

No, Vitess is more along the lines of our Gizzard system which has been out there for a while:

https://github.com/twitter/gizzard

Reply on news.ycombinator.com