Hacker News Reader

Comment by kolinko | Hacker News Reader

Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines?

Right now it is extremely difficult to build your own web crawler that would compete with Google. And that is not because of the technology, but because multiple sites will prevent your bot from accessing them if you're not Google or Bing - either through robots.txt, or through directly banning your IP if it's trying to crawl and it's not a confirmed google-bot.

Having a non-profit, open source, crawler that keeps an up to date index (or web cache) of the web would help competition spring up.

View on HN

Why is https://commoncrawl.org/ not enough?

Isn't that one more data sets for ML other research purposes instead of a highly up to date search index (For example with news from a few minutes ago).

Roughly speaking, yep - Common Crawl provides a sizable chunk of web data (420 TiB uncompressed, over 3 billion unique URLs, as of May 2022; historic statistics here[1]), and is updated on monthly basis. Not near-real-time, true, albeit relatively fresh.

A question to ask could be: how often do users care about information from a few minutes ago, compared to information that has been available for a longer duration of time?

[1] - https://commoncrawl.github.io/cc-crawl-statistics/

Isn't that more a question of adding to the mix frequent scraping of

- a few thousand news-sites (like nyt.com, bbc.co.uk),

- a few thousand very popular blogs (based on what influencers people search for),

- a handful of social media sites (e.g. Twitter),

- a few hundred databases in areas like weather, airlines, sports (like ATP for people who look for Wimbledon results today)?

I mean, any time someone wants information on current or recent events is your use case right there. If you exclude news entirely, you could maybe disregard recent websites but I imagine that's statistically a pretty large portion of search.

I built something with an API that uses Selenium to image a site. It works on a large percentage of the sites I feed it.

I don't recursively call links found in the pages. I expect the user to give me the URLs to crawl and save.

In order to "find" new content, I let the user specify where they want to search for things the engine hasn't "crawled" yet. So, a search for scooters to buy might end up searching Amazon directly, then lets the user "save" the site by passing the Amazon URL for the scooter they like to the system for imaging.

I use GPT-3 or other ML models to do some of the heavy lifting for adding labels to the pages or documents the user uploads.

This ends up being a "curated" list of documents important to the individual user, not an exhaustive crawl of all things which are important to all users.

> Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines?

That's a excellent idea! In the spirit of open-data, and people can do with it what they want.

I think this is a great idea. How does this work with copyright? Search engines seem to be able to download a reproduce content from scraped pages (and wrap it in ads, and derive content from it) this is called “indexing” when they do it but scraping when everyone else does it.

> and wrap it in ads, and derive content from it

i am probably missing something but can you give an example where this happens?

E.g. on Google if you search for "how to tie a tie", a little info box may pop up with step by step instructions. This content is taken from some website, but that website gets no page hits or ad revenue. Instead, Google gets to serve ads on the search engine results page.

(I don't know if this happens for this specific example, but Google does this for some searches)

> that website gets no page hits

Part of why sites participate in the infobox program is that in practice you do get quite a lot of hits from it: many people click through to see the answer in context.

Ok, I just tried but don't see that info box but that is exactly what i was asking for. Thank you very much, did not think of that.

I think they're referring to how Google "extracts" answers from your website and shows it on the search results page. Effectively meaning that the user doesn't even need to go to your site to get the answer, because Google extracted it and gave it to them directly.

It seems to me that what they usually extract is some junk only vaguely related to the query and often cut apart and reassembled in a way that's just wrong.

> Right now it is extremely difficult to build your own web crawler that would compete with Google.

Nah, not a small task but you can break it down into well understood problems that have known solution.

The hard part is ranking everything.

Are you speaking from theory or experience?

If from experience - how did you get around multiple sites disallowing crawlers other than google or bing?

Yes, the article could better read, 'Why the world needs a range of non-profit search engines'

The data is available! Not currently documented but check out the code for the API.

I'm curious if you think a co-op would be feasible between Mwmbl and other like-minded crawlers who are interested in taking a divide-and-conquer approach to crawling the web.

> Perhaps a better approach would be building an open source www index...

Wouldn't Google just use this too? Which would give Google in greater dominance over alternatives...

Reply on news.ycombinator.com