Hacker News Reader

Comment by p-e-w | Hacker News Reader

I like the idea of searching a curated list of domains, but I'm not sure that doing the curation yourself is the best approach considering the huge number of useful but niche websites in existence.

I wonder if simply parsing all of Wikipedia (dumps are available, and so are parsers capable of handling them) and building a list of all domains used in external links would do the trick. Wikipedia already has substantial quality control mechanisms, and the resulting list should be essentially free of blog spam and other low-quality content. Wikipedia also maintains "official website" links for many topics, which can be obtained by parsing the infobox from the article.

View on HN

Oh, and more to the point: This is a role Wikipedia explicitly renounced, isn't it. When it became so big and the Google PageRank gave it high importance, the spam became unbearable so Wikipedia decided it needs to change the incentives and it applied the rel=nofollow to all external links, so that it could stop working as an unpaid manual spam filter for the whole internet. Sure, your new search might ignore the rel=nofollow but if you ever become big enough, the incentives of spammers would lead to a bigger spamming pressure on Wikipedia...

That's easily fixed by relying only on protected or high-profile pages. Those already deal (mostly successfully) with spam and NPOV violations on a daily basis, so piggybacking on those protection mechanisms should yield a fairly high-quality pool of curated external links.

While that sounds good in theory, who is to say those that can edit protected and high profile pages aren't in SEO spammers' pockets?

I mean the edit history is public and there's plenty of people that actually pay attention to edits and the like so they would be found out soon enough, but still.

I'm sure this is an ongoing discussion when e.g. political figures' pages are protected as well - who becomes the gatekeeper, and what is their political angle?

Sure, but at that point you're simply discussing Wikipedia's quality control system, which may be an interesting discussion, but has nothing to do with search engines per se.

Considering that Wikipedia has become a pillar of most scientific work (imagine writing a math or computer science paper without Wikipedia – utterly unthinkable), it's safe to say that knowledgeable people have collectively decided that its quality control mechanisms are "good enough", or at least better than those of any other resource of comparable depth and breadth.

And that puts Wikipedia's link pool leaps and bounds ahead of whatever dark magic current search engines are using, which mostly seems to be "funnel the entire web through some (easily gamed) heuristic".

Ehh, I think you’ve overestimated the quality of Wikipedia on highly specialist topics (like, say, the kinds of things you’d write academic papers on). It’s not so much that it doesn’t have the content, it’s that the coverage is super uneven; sometimes it has extremely detailed articles on a niche theorem in a particular field, and other times it has the barest stub of information on an entire sub-sub-field of study.

The "official website" links can be retrieved in a machine-readable way from Wikidata. E.g. a completely random silly example of official websites of embassies: https://w.wiki/5T3R

Is there documentation for rolling your own?

I've been considering building my own search engine for a while for my niche topic which has <50 websites and blogs on the web.

I can't tell how useful this will be but it'd be fun to give it a go.

Elastic App Search would be well suited for something like that. It comes with a built in crawler.

Your own what? Your own query? Sure; it's just a SPARQL query over the Wikidata data model. The documentation portal for the query service is at https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/... though you'd need some familiarity with Wikidata, its properties, etc. E.g. the "wdt:P856" in my query is the "official website" property on Wikidata: https://m.wikidata.org/wiki/Property:P856

Only problem is that Wikidata is still incomplete when compared to Wikipedia itself. But yeah it's "trivial" to search it.

That's our strategy at you.com - we start with the most popular sites, crawl them and build apps for them, eg. you.com/apps and let users vote in search results.

Full disclosure: I'm the CEO.

> you.com/apps and let users vote

what is stopping companies/users from abusing/gaming on this system with bots?

Richard, CEO at you.com here.

Users vote on their own preferences... not on other users' preferences. so i dont see how it will be gamed unless we start incorporating user preferences into global ones - which we would only do if we thought it worked better and isn't being gamed.

Your search actually performed better than Google for me on a random query. I queried both engines "What is the specific heat of alcohol?", Google threw up a rich search answer that linked to some random Mexican site that is clearly exploiting SEO [1], you.com linked me to answers.com (which is more trustworthy than random mexican website).

[1]: http://elempresario.mx/sites/default/files/scith/specific-he...

When I search Google I get a rich search answer of http://hyperphysics.phy-astr.gsu.edu/hbase/Tables/sphtt.html

I just learned about you.com from this thread. It looks very promising.

I've looked into this, and found wikipedia's links not to be super useful. Wikipedia prefers references that don't change, so books primarily, and beyond that websites that don't change, so academic journals where you're getting paywalled, and WaybackMachine archives of websites (even if they are still live). You aren't getting much use out of wikipedia.

The problem with only using curated lists is that you kill discoverability of new sites, but it does have promise like we've seen with Braves 'goggles'.

Reply on news.ycombinator.com