248 karma · joined August 13, 2009
Github: https://github.com/chrisgoffinet ThreeComma: https://threecomma.io
The OS usually does a pretty good job, but remember it's designed to be generic not for a specific use case. When you really care about performance of a database you tend to bypass what the OS does, and take control bypassing things like the scheduler, cache, etc.
Over time, Manhattan evolved into a fully fledged read/write database that is able to support batch + read/write in the same system. Batch is great for some use cases, but sometimes the cost is too expensive when you factor in how much processing power, storage, etc is needed for that model. We support both of course still we want developers to have the freedom to increase their productivity.
http://chrisgoffinet.com/pin_network.sh
This will set queues 0-7 to specific smp affinity slots.
http://www.metabrew.com/article/a-million-user-comet-applica...
Part 3 is my favorite. http://www.metabrew.com/article/a-million-user-comet-applica...
http://www.slideshare.net/nkallen/q-con-3770885
Now, at the time they were doing peak 2000 tweet/s. The fan-out was 1.2M deliveries a second... So if we go with the current 600:1 ratio at 7000/s, that's about 4.2M/s. I actually know it's much higher now since I work there but other things to consider is we have a large data warehouse, search, API, pipelines to external parties for the firehose, logging at terabytes an hour, in-house metric collection doing 3M writes/s, etc.
http://www.scribd.com/doc/59830692/Cassandra-at-Twitter
It add's up very fast.