For incredible scale, and heretic ideas, I think people should consider the decisions that Colin Steele made as CTO of RoomKey. He was thinking about how to build a database that could offer nearly real time searching and yet handle any level of traffic. The idea he hit upon was compiling his code with a static snapshot of his database. Think about this carefully: instead of lots of webservers putting strain on a central database, every web server has its own snapshot of the database. If traffic doubles, you can double the number of instances you have on Amazon, and you are doubling both the number of web servers you have, and also the number of databases you have. The team at RoomKey has built this app with Clojure, and stores the data in an instance of Solr that actually gets compiled with the code. With an automated build system, you could probably roll out new snapshots of the database several times a day, so the data is never very stale.

This is how Colin Steele wrote about it on his blog:

"Put another way, users of this system have a high tolerance for inconsistent reads. Bob’s and Jane’s hotel universes need not be identical. (They can’t be completely divergent; eventual consistency is fine.)

So: A-ha! The messy relational data could live in a secluded back-end “content sausage factory”, whose sole purpose in life would be to produce a crisp, non-relational version of the hotel universe as known best at that point in time.

This “golden master”, non-relational database of hotels could then be shipped off to the live, operational system which faces users.

Moreover, different users might be exposed to different versions of the “golden master” hotel database, allowing us to test and to do progressive and continuous rollouts. Decision One: I put relational data on one side and “static”, non-relational data on the other, with a big wall of verification process between them.

This led to Decision Two. Because the data set is small, we can “bake in” the entire content database into a version of our software. Yep, you read that right. We build our software with an embedded instance of Solr and we take the normalized, cleansed, non-relational database of hotel inventory, and jam that in as well, when we package up the application for deployment.

Egads, Colin! That’s wrong! Data is data and code is code!

We earn several benefits from this unorthodox choice. First, we eliminate a significant point of failure - a mismatch between code and data. Any version of software is absolutely, positively known to work, even fetched off of disk years later, regardless of what godawful changes have been made to our content database in the meantime. Deployment and configuration management for differing environments becomes trivial.

Second, we achieve horizontal shared-nothing scalabilty in our user-facing layer. That’s kinda huge. Really huge. One of our crack developers and now our Front-End Development Manager drove Decision Three. Our user-facing servers render JSON. A Javascript application, bootstrapped by a static HTML page, creates the entire UI, rendering elements into the brower’s DOM as it interacts with our API. This so-called “fat client” or “single page Javascript app” has been steadily coming into vogue for the last few years, but it was far from an obvious choice at the time.

We reap several benefits from this decision. First, our UI gets faster without us having to pay for new servers that take advantage of Moore’s Law. Second, our UI gets faster because browser makers are dumping tons of time an energy into improved JS performance. Third, HTML is verbose and JSON isn’t, saving us both compute cycles and bandwidth on the server side.

These decisions en toto yield a picture of a three-layer cake, with a messy relational tier on the bottom, an operational API and SOA tier in the middle, and a fat client presentation layer on top."

http://www.colinsteele.org/post/27929539434/60-000-growth-in...