I don't think you have a choice, you'll have to use MapReduce if the data is too big to fit into memory. I believe that's what Facebook and Google do:

https://www.facebook.com/notes/facebook-engineering/scaling-...

http://googleresearch.blogspot.com/2009/06/large-scale-graph...