Google and Facebook Team Up to Modernize Old-School Databases | Hacker News Reader

Google and Facebook Team Up to Modernize Old-School Databases

wired.com

48 pointssalemh25 comments

So relational databases are old school, and column stores are all the hip trendy thing, despite throwing away a ton of features?

Well, there are three types of databases: Relational, Columnar, and finally massively distributed. They throw away different feature sets in order to handle a certain subset better.

The primary reason Relational databases are so friggin' strong is that they have a solid foundation and maturity. Most new "NoSQL" stuff is immature crap which should never be used in production for any kind of persistent data. But they are being used as such, and I do note that matures the products over time.

Tell me about it. I have been hauled into a project, where we are reimplementing a relational database using NoSQL. No idea why, other than the fact that they want to use NoSQL.

We are now clearly in a world where many technology journalists are entirely unaware of the history of computing before the year 2,000 or so.

Worse, we are now clearly in a world where many developers are entirely unaware of the history of computing before the year 2,000 or so.

I mean, we had "NoSQL" before we had SQL. Tons of schema-less, hierarchical etc DBs, talking directly to your program and such. Anybody that uses them without massive scaling needs doesn't know his history.

Relational databases are not the right choice for large-scale data stores serving large number of users because:

- they provide unnecessary features at high cost (e.g. database-wide transactions),

- they don't provide features that are essential (e.g. scalability, distribution, graceful degradation).

By contrast NoSQL data stores like Bigtable are highly scalable and can easily be stacked over to provide more features if needed (see e.g. Megastore, http://pdos.csail.mit.edu/6.824-2011/papers/jbaker-megastore...)

What is the rationale of putting this work into Mysql? With its sloppy execution and the dark clouds of Oracle hanging over it? Why not Postgres, or even MariaDB instead?

Because they are locked in to Mysql and can't easily switch to something better.

I'd really like to see the evidence for that. Postres > MySQL, and MySQL doesn't have that many lock-in features last time I looked.

Any Google/FB/Twitter employees here than can elaborate?

(I am a manager of the Data-Perf team at FB which deals with lots of different DB technologies)

MySQL (specifically InnoDB) is extremely efficient as a storage backend compared to PostgreSQL.

There are a few features that make InnoDB better in many cases:

1. Change buffering for IO bound workloads: If you are IO bound, then InnoDB change buffering is a huge, huge win. It basically is able to reduce IO required for secondary index maintenance by a huge amount.

http://dev.mysql.com/doc/innodb/1.1/en/innodb-performance-ch...

2. InnoDB compression: When you are space constrained (say using flash storage), then being able to compress your data is a big win. In our case, it reduces space by around 40% which translates directly to 40% less servers required. While you could do something like run PG on ZFS with compression, for an OLTP workload, you want the compression in the DB so that it can do a lot of work to minimize the compressing and decompressing of data.

https://dev.mysql.com/doc/refman/5.6/en/innodb-compression-i...

3. Clustered index: The InnoDB PK is a clustered index. This makes a lot of query patterns (such as range scans of the PK) very cheap. Combined with covering indexes (which PG now has too!), you can really minimize the IO required by properly tuned queries.

There are a variety of smaller things as well, such as InnoDB doing logical writes to the redo log vs. PG doing full page writes, so on very high write systems, the REDO log bytes written will be dramatically less. Also MySQL replication has traditionally been more flexible than PG, but PG has made some great strides recently, so I don't know if I would maintain that position still.

Thanks a lot for the detailed reply, this is why I love HN!

1-2 years back I saw a benchmark stating that PG works better than MySQL on I/O constrained AWS instances, so I didn't expect this ("I've learned something today").

I think Postgres is a better default choice because you are likely to be safer and for it to behave properly with your data. In certain cases, apparently Mysql might be faster for certain workloads, and with certain tradeoffs. That's the kind of informed decision you can make once you're scaling to that size.

Thanks. Presumably these optimizations could be added to postgres as well, no?

Since work is/will have to be done to the chosen db, it would be best for everyone if the one chosen was of high integrity.

There is nothing that prevents these from being implemented for PostgreSQL. In fact, they have already done a lot of optimizations that close the gap, such as covering indexes. Before covering indexes, InnoDB was even further ahead of them.

Of the ones mentioned, I would guess compression is the most complex. For InnoDB this took many years to get the point of it being usable and efficient. Many naive implementations can cause huge overheads for CPU which makes it unusable.

Your smaller points don't make much sense to me... PG doesn't have a REDO log, that's just not the way it's architected. If you mean WAL, a patch went out in 9.4 to prevent updates to a page from rewriting more than necessary, so they're not doing full-page writes each time. Clustered index also makes inserts slower--it's again not a straight win for MySQL (and what query patterns other than range scans of the PK does it make cheaper?). Finally, while obviously it's not the same as InnoDB compression, TOAST does a rather good job in practice, and Postgres's indexes are quite efficiently compressed (especially with new changes in 9.4). I can't speak to (1), but it's not at all clear to me that any of these advantages put MySQL ahead in the long term, and certainly not for all workloads.

Right, PG calls the REDO log the WAL (for most purposes they are the same thing). I did not know that 9.4 can do partial page writes to the WAL now. Guess I will have more reading to do, thanks for pointing it out! A nice blog post by a colleague recently showing how large writes to redo logs matter is (not about PG, but why it is significant in the context of size of entries):

http://smalldatum.blogspot.com/2014/03/redo-logs-in-mongodb-...

As far as when clustering a table is useful, see the CLUSTER command in PG. It is roughly the same places you would want to do it, except it is automatically maintained. You do need to realize what is going on to minimize impact on inserts, but in a lot of cases, data in inserted in generally ascending order so it mostly 'free'. This does make GUIDs really bad for PKs in InnoDB.

Clustered indexes are like covering indexes (which PG got recently). You don't quite realize how useful they are until you get access to it ;)

InnoDB compression for us is primarily for non-lob objects, so TOAST is quite a bit different than the cross-row compression that we get. We will normally do compression of large objects outside of the DB whenever possible.

I'm not saying that InnoDB is always better than PG, but in a lot of cases that I have tested it with, it is indeed better. PG has come a long way recently, including options such as covering indexes to close the gap.

Sounds like a fork of MySQL, which makes it similar to MariaDB (Note that Google's performance patches for MySQL are similar already).

Facebook has previously answered this question on Quora: http://www.quora.com/MySQL/Is-Facebook-considering-ditching-...

Also from the WebScaleSQL FAQ:

Q: Why didn't you base this on MariaDB, Percona Server, Drizzle, etc.... A: We reached a consensus that MySQL-5.6 was the right choice for this, as it has the production-ready features we need to operate at scale, and the features planned for MySQL-5.7 seem like a fitting path forward for us. We will continue to revisit this decision as the ecosystem evolves.

I don't think "Webscale" is the right word; they should have used "Worldscale". The web part is irrelevant here.

It's a joke referring to the "MongoDB is Web Scale" meme. The FAQ is quite suggestive:

Q: Why is it called WebScaleSQL? A: While there are a variety of origin stories for the name depending on who you ask, ...

They just don't want to mention it directly. :)

Every time I see the word "Webscale" I think of this animation:

http://www.youtube.com/watch?v=b2F-DItXtZs

The name was SPECIFICALLY picked to refer to that animation, it's an inside joke kind of thing.

Previous discussion in HN at least here: https://news.ycombinator.com/item?id=7480843

Project page: http://webscalesql.org/

Currently under what circumstances will we see performance increases, and by how much?

Well not just Google / Facebook but Linkedin and Twitter are also in the team.

Reply on news.ycombinator.com