I applaud the effort, but the search results quality is really poor. It's basically an Altavista style laundry list of anything remotely relevant in no particular order.
That said, do you do your own crawling or do you source results from someone?
That said, do you do your own crawling or do you source results from someone?
The incoming data to the crawler... not so sure... something on the order of 10-20tb. I didn't really measure this. And I don't keep all the data. There is no "cache" function.
On a 1gbit/s connection it takes about a week to crawl and generate the index.