High Concurrency MySQL @ Facebook #mysqlconf

It seems like the primary concern for Domas (presenter) is concurrency within InnoDB. Just to get it out of the way, he dismissed MyISAM, and went right into the low level optimizations & tweaks they're working on at Facebook. High level points are:

- they're using XFS, and he stressed understanding the block math and how it relates to your record sizes, etc.

- using "correct" scheduler has a big impact

- swappiness = 0, etc.

For InnoDB itself, it seems like they're carving out entire chunks of functionality that he claims "never worked correctly" anyway. For example, deadlock detection is something they disabled outright because it gave them a large perf boost. Likewise, for read-ahead functionality within InnoDB. Those were the two he went into a lot of details.

Concurrency + threads: there is no definitive formula, but he suggested setting thread concurrency to be the min concurrency of your lightweight requests (claimed: threads are cheap).

For ops / day-to-day, they focus on slow servers and dive right in: custom tools + gdb + freeze the server. They are playing with the idea of providing stored procedures interface to app developers since it'll avoid the round trip time between different queries (TBD). Also, they sometimes go very aggressive (timeouts of 3s, or even 1s), to force internal app devs to respect the DB layer.

Also hinted that they're using "smart caches" which are capable of buffering counters / updates and doing one update instead.

All in all, their biggest problems seems to be thousands of (app) servers that are always connected, hence the need for high concurrency...

(I'm sure I'm forgetting a whole lot, but that's off the top of my head)