Hacker News Reader

Comment by Xeoncross | Hacker News Reader

Plug for https://commoncrawl.org/ if you need billions of pages but don't want to deal with scraping the web yourself.

It would thrill me if common crawl were updated with such frequency that it would allow new search engines to enter the market

I haven't dug into it enough to know if there's some technical reason it's not currently the case, or just lack of (interest|willpower)

I'd argue that one broad crawl every 2-3 months in addition to their updated-daily news crawl[0] should be good enough to make a rudimentary search engine.

[0] https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html

You're right about the "rudimentary" part, because I don't know how they do it but the major players have some not-kidding-around freshness:

https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%2...

https://www.bing.com/search?q=%22thrill+me%22+%22common+craw... (and DDG similarly, because bing)

ed: I was curious if maybe HN publishes a sitemap, and it seems no. Then again, hnreplies knows about the HN API so maybe it's special-cased by the big crawlers https://github.com/ggerganov/hnreplies#hnreplies

I think it is as simple as an algorithm that rates how frequently new updates show up vs how many people visit a site.

If you rarely make updates to your site, Google crawls it infrequently and new things won't show up very quickly.

But if you do have frequent updates and lots of traffic, like any popular forum style site, you will get lots of crawler traffic. And I would bet the algo does the same for all endpoints on a site. So the "about us" page on a popular site probably ends up not being crawled nearly as much as the new threads page.

Is there subsets of common crawl anywhere for individual sites. E.g. YouTube for example?

You can query subset of specific sites from common crawl itself.

Reply on news.ycombinator.com