I huge Lucene index for storage and search, with a custom crawler that I wrote myself. It's a fun engineering problem.