Comment by weinzierl | Hacker News Reader

Comment by weinzierl | Hacker News Reader

Isn't this what Common Crawl[1] is. From their FAQ:

> What is Common Crawl?

> Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis.

> What can you do with a copy of the web?

> The possibilities are endless, but people have used the data to improve language translation software, predict trends, track the disease propagation and much more.

> Can’t Google or Microsoft just do that?

>Our goal is to democratize the data so everyone, not just big companies, can do high quality research and analysis.

Also DuckDuckGo founder Gabriel Weinberg expressed the sentiment that the index should be separate from the search engine many years ago:

> Our approach was to treat the “copy the Internet” part as the commodity. You could get it from multiple places. When I started, Google, Yahoo, Yandex and Microsoft were all building indexes. We focused on doing things the other guys couldn’t do. [2]

From what I remember reading once DuckDuckGo doesn't use Common Crawl though.

[1] https://commoncrawl.org/

[2] https://www.japantimes.co.jp/news/2013/07/28/business/duckdu...

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

Would users notice for many searches? Obviously it wouldn't be useful for news or social media, but for practically everything else a latency of a month would be fine.

Yes. As the Internet Archive shows, a large corpus of valuable content is no longer changing.

It absolutely breaks any use for information produced in the last month. Here's a few things that come to mind:

While you say "news" really that covers any information about current-ish events. It's not just "what happened today" but background on things like the Muller report right now.

Any technology release, or update.

Reviews of any hardware or software.

Information about security vulnerabilities.

Film reviews.

Game reviews.

Book reviews.

New scientific publications.

Books, games, hardware etc are all sold for several years.

What you’re describing is just news on the latest updates representing a small slice of the market. Making the remander far from useless.

I don't know this for certain but I strongly suspect news (and reviews/criticism, which is editorialised news) doesn't make up a huge percentage of search traffic. People read news sites that align with their preexisting points of view. They don't often go looking for new perspectives. If someone wants to know what's happening they want the filter of their preferred news outlet, if they want a review they want to read or watch their preferred reviewer. They don't want whatever happens to be the top search result.

Although, that said, with Google personalising search results the top result is very likely to be the user's preferred site anyway. We can't have people seeing outside their filter bubble after all.

For many use cases I would imagine that an index that was a bit delayed might actually be preferred. I'm not entirely sure what you meant to imply by 'research purposes' but many of the use cases I imagine are scholarly use cases where something a more stable would be preferable. That said I seem to recall Henry Thompson telling a story about trying to do a study of the statistics of the net using common crawl. By the time he was done he ended up being less certain of the results, the understanding, and the methodological validity of anything related to trying to measure the internet by looking at a single snapshot of a subset of the link structure. Too hard to understand what you are actually counting.

edit: yep here it is https://doi.org/10.1145/3184558.3191636

Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with your own more regular crawls.

I'm curious how they track and rank a site's "change velocity" without crawling all of the internet all of the time. It almost seems like a catch 22 no? Might you have any insight into how this works? Any suggested reading or links?

A site you just learned about probably isn't very important, so you can just watch its month to month change. You don't care if they change that much because they aren't important yet.

A site that gets lots of links quickly (and is therefore important) will likely garner them from sites you are already frequently visiting.

Apart from that Common Crawl respects robots.txt (which makes sense) so many sites you expect to see there are not indexed. Netflix, Facebook LinkedIn and many more. If common-crawl sees serious adoption those sites will modify their robots.txt but it's and chicken/egg problem.

There is a simple solution: if companies do not respect do-not-track then why should we respect robots.txt?

Because then you end up in an arms race that the little guy usually does not win.

There are a significant number of crawlers out there that don't respect robots.txt. The usual response to them isn't to roll over dead, it's to get CloudFlare (on the technological end) and/or sic the lawyers on them (for CFAA, IP, or ToS violations).

This is literally why I created my company:

http://www.datastreamer.io/

We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C.

We're not free but the primary reason we exist is that other vendors charge borderline extortionate pricing and I fundamentally believe that the web MUST remain open.

We've also been providing data for very affordable pricing to researchers for more than a decade.

Search for us as Spinn3r under Google Scholar (our previous name) and we have hundreds and hundreds of PhDs who have access to our data.

We do charge for research usage now but it's very very very affordable.

The entire point is that we're trying to enable innovation.

This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.

You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope these indices are available on AWS, Azure etc where people can just use it with cloud compute and pay per use.

> I don’t think web index would ever be free

Yet the company first mentioned does it for free, lol:

https://commoncrawl.org/

I've checked Datastreamer.io for 5 seconds, I don't see any link to their repo. If not "open source" then what does "open" mean?

Commoncrawl is not a company, it's a non-profit. Open means you can access the data, there is no assumption about the data being free or not.

What? It's a nonprofit organization engaging in nonprofit business. Any organization that engages in business is a "company." Common Crawl is a company. Your comment isn't accurate and it doesn't address the parent's comment.

Easy to test, though. If they were open, you could download their entire data set under some permissive license. If you can't then they are not open.

Cheaper than the competition? Maybe. Nothing that requires contact to get a price is "affordable" (if you have to ask, you can't afford it...)

Have you considered making a subset of your data open, cross-referenced from the paid data set? If other providers followed this approach, the open data set could grow and become more useful to all of the paid data providers, if only for lead generation and tool interoperability.

How exactly is it open if you have a paywall blocking people from accessing it though?

If your prices are so much more reasonable than competition, why are they not published publicly on your site? “Contact us and we’ll tell you the price” is shady for a service that claims to be “very very very affordable.”

Because they charge different rates to different people. Super common in b2b arrangements.

Hacker News guidelines say:

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize.

Your comment makes zero sense in this context, because it's just marketing.

> we're trying to enable innovation.

You're trying to make profit, like every other company in the world and that's OK.

Yup, article says "A search engine needs to keep its index current, meaning it needs to update at least a part of it every minute. This is an important requirement that is not being met by any of the current projects (like Common Crawl) aiming at indexing snapshots of (parts of) the Web."

Common Crawl is referenced in the document.

I don't think it's very democratic if it's only hosted on Amazon S3. Effectively, this gives Amazon control over the data.

Reply on news.ycombinator.com