Hacker News Reader

Comment by mark_l_watson | Hacker News Reader

A good reason to use and support Common Crawl - a good resource, that is updated (I think) every few months).

Cuil (ha!) donated most of their crawled data to the Internet Archive - so with that plus Common Crawl, it's pretty easy for anyone to have a sizeable index.

What those neither of those give you is an up to date index, which is why small search engines still need Yahoo and Yandex's APIs. I'm not sure any free resource can match the speed at which big companies can index the web.

Common Crawl is more interesting for research purposes than for real search engine business. It refreshes only 4 times a year and has only 2 billion pages. For anything under ~10B pages is most likely to be called "research" or "toy" index. You have almost zero chance of building competitive index if you are below ~15B. Google is rumored to crawl about 150B pages as reference. There are estimated 1.2T pages out there which does not get crawled due to various reasons.

I dont think that you need so much pages - most of 150B pages will be never ever shown displayed at SERP.

True, you just need a subset. Now how you you identify that subset without indexing the pages to find out whether each page is in the subset you need?

IIRC google used to scan different pages at very different frequencies. Quite possibly because it has assigns pages into subsets every time it indexes.

Perhaps. The problem is that crawling the web isn't that hard. Foreach(links) { get(link)} is all you really need. Doing it multiple times is harder though.

The real problem imho is building a distributed index and fighting spam. Both are incredibly hard problems to do well and very expensive. Hence so few are trying to do it.

If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.

You say that like it doesn't happen all the time. googlebot included.

I think providing quality results is the challenge. That take certain algorithms and optimizations to do so in a timely fashion. This is where AI projects like Watson really come into play add they are all about retrieving the correct information (a difficult problem on its own) in a timely fashion.

Also true, however I think a basic ranking algorithm should still be fine for most cases if you have 100% spam eradication. Good content left over should bubble to the top.

I tried building a small search engine with a very basic algorithm and it worked very well for 90% of searches.

I agree with you in part but I also believe if you got the others 100% right you would have a respectable engine by itself.

Reply on news.ycombinator.com