Using hadoop for batch processing and cassandra to serve real time data are among my tools of choice. chef_cluster looks like a nice bag of tools for setting up and managing a scale-free computing infrastructure.