Hacker News Reader

Comment by aheilbut | Hacker News Reader

I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.

View on HN

A significant problem with this is trust. You can't trust websites to reliably or accurately index their sites due to both incompetence and malice. I don't think there's any way around the malicious component. Formal or informal standards may take care of the competence factor with the feature being built into common publishing platforms.

XML sitemaps are a microcosm of putting the indexing onus on websites instead of the search engines - they are basically ignored by search engines because they have been abused and are not a useful signal. If pages aren't important enough to be linked to throughout your website then they aren't interpreted as being important enough to return to users. The optimistic case is that sitemaps/indices will send parallel signals to the search engines in which case they are redundant. The pessimistic case is that the sitemaps/indices will send signals orthogonal to the content provided to users in which case the website is either being deceitful or incompetent. In any case, the search engine will not want to use the sitemap/index as a signal as it either doesn't provide value or provides negative value.

The code for doing the indexing (at least by default) could be built right in to the web server, so it'd just be a matter of enabling an option in Apache or the like.

It would be pretty easy to verify whether or not the index is accurate with a small random sample of pages on the site, and then penalize / exclude (or do a de-novo crawl) for those sites not providing a legit index.

One solution could be to have a verifiable module on servers that creates and serves the sitemap. Something like a signed certificate or DRM.

Honestly I'd much rather have a bunch of dice rolls on incompetence than the current centralized, single point of control over the entire index.

Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using. It's difficult to imagine a worse solution for the web than this current model.

"Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using."

Wow, interesting, this is the first I've heard of this. Might you have some link or citations about this? Thanks.

4% of the Google index hit by de-indexing

https://searchengineland.com/4-of-the-google-index-hit-by-de...

I think there are a few factors that would make this idea unworkable. There are two categories of issues, technical and economic, that prevent this from working. I'll go into more detail about the technical issues.

The top-level problem is fan-out. If you want to fan the query to the top million domains (far too few to match Google's retrieval depth, but enough to demonstrate the issue), you're going to need to implement some sort of multi-level fanout, since just one machine can't send enough HTTP requests -- nor even establish that many connections -- in a reasonable time-frame. There are going to be severe tail latency issues that will prevent you from gathering documents from potentially relevant sites. You will have to make frustrating tradeoffs about when to time-out per domain queries to provide a good user experience. And many more issues besides. All decisions that are unnecessary if you control the index. Also, internet bandwidth isn't cheap, and you're going to need a lot of it just to consume the top ten results from a million sites.

The next technical issue is that the inverted index is only a small part of what goes into information retrieval. Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data. Then they do a second pass on a more comprehensive form (i.e. the whole document, and metadata about the document), but over fewer documents. There can even be third and fourth passes. The data for the first pass is often embedded directly into the index, and it would be challenging to come to any kind of agreement among stakeholders about what data should be embedded. This goes double for the second and subsequent passes. Moreover, those second and subsequent passes often use data about the document, rather than data in the document. Data a site owner would be unable to provide or even incentivized to falsify. Not less than these problems is the issue of where to run the scorer code. If you're running it locally, you're operationally 90% of the way to the complexity of an inverted index. Why not go all the way?

Then of course there are the economic issues, which, roughly, are: "Why should I pay all this money to host an index of my site that nobody uses when Google will do it for free and charge me nothing?"

>"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data."

Can you elaborate on what is the "simple correlated representation of the data"? It sounds like you understand this space pretty well might you have any links or literature on how modern crawling architectures and indexing work? Thanks.

Sorry, that's just a complicated way of saying that you can embed data in an inverted index that lets you guess how likely a document is to be found relevant on subsequent passes. Basically, you use various properties (embedded in the index) to filter down the list of documents you want to inspect in more detail, as you perform retrieval. There is some information in [1] about some types of filtering that can be done (e.g. their discussion on tiered retrieval and the notion of a global quality score being used to discard candidates). Lucene calls these bits of index-embedded data "token attributes," [2] but how exactly they are used depends on the scorer implementation. To learn more about how the industry approaches these issues, unfortunately you have to join one of the companies that's on the leading edge of this type of research, since they are loathe to disclose too much.

[1]: https://nlp.stanford.edu/IR-book/pdf/07system.pdf

[2]: https://pdfs.semanticscholar.org/2795/d9d165607b5ad6d8b97183...

Thanks for the detailed explanation and the links, these are really helpful. Cheers.

PubSubHubbub was intended to be something like that.

https://en.wikipedia.org/wiki/WebSub

Like a sitemap? https://en.m.wikipedia.org/wiki/Sitemaps

That is a start, but I mean an actual inverted index, and preferably even more structured indices with appropriate metadata. Web servers should also be responsible for archiving themselves and providing a change history.

No, an actual search index, offerring word-to-URL mappings, along with metadata: creation and revision dates, authors, titles, file and/or MIME types, URLs within the text, other attributes, etc, from which a search interface could query.

Page-ranking would remain an issue, likely outside this scope.

I'd like to see some sort of cache-and-forward structure.

And you'd be relying on good-faith actors, which means heavily penalising bad actors.

This reminds me of how DNS works. Every domain holder is responsible for their nameserver records but every dns server ultimately communicates with a distributed network for lookups. Blockchain would be a great solution for this.

Reply on news.ycombinator.com