Hacker News Reader

Comment by desertraven | Hacker News Reader

I’m working on a search engine crawled by users themselves. If I think about it, I'm not interested in 99% of the web. Therefore I don't need to deal with exhaustively crawling the entire web.

Hypothetically, if I could access only the sites visited by the HN crowd, that would be fine by me. Somebody in an art community might feel similarly. If there were some metric the crawlers could use to rate a page (time spent on page, manually downvote, etc), then the signal/noise ratio would be really good.

Obviously if I wanted to do a really broad search, I'd use a typical search engine, but if it's tech related, I wouldn't mind pages that have been browsed by HN.

View on HN

I'd be pretty happy with a search engine that lived on my local machine, just for pages I've already visited.

That might be a good first step towards what you're planning. Step two would then be figuring out how to do federated queries across users with similar interests.

From a privacy standpoint, I'd be less uncomfortable letting random strangers run (sanitized) queries against my browsing history (stored locally), than I would be uploading my browser history to a third party.

Great points, thank you. I've been wondering how to do it in a privacy-preserving way. Also the current implementation is a browser extension, I wonder if there may be a better architecture.

Can you point to the extension for people to test

Are you old enough to remember https://en.wikipedia.org/wiki/Webring ?

Take that idea and evolve it into a federated search service. If your site is in the "fishing" ring then your pages get indexed into the fishing index and the search widget on your site would potentially return results from all the sites in the same ring.

I guess I want SearchRing.

YaCy [1] already exists, but I've never tried it. I've been wanting to set it up (locally at first, not participating in the decentralized network, unless I really like it) to crawl only sites I choose to add to the list – I was thinking perhaps just blogs and documentation that I like, along with anything I bookmark.

That being said, I think YaCy looks dated, the website has several typos, and I get the impression that not many people are still active in the network. I feel that a new project with similar technology in a "hotter" language (just for popularity's sake) could get a lot more traction.

[1] https://yacy.net/

You want something where users voluntarily (and anonymously) submit their search histories and they can be compared? That sounds interesting.

>if it's tech related

https://notrashsearch.github.io/about.html might help (https://news.ycombinator.com/item?id=29774456)

There's also https://search.marginalia.nu/ (https://news.ycombinator.com/item?id=28550764)

Are there more compelling use-cases than creating public-good kind of whitelists? What incentives encourage sharing of websites? I can list some undesirable incentives: marketing, "shilling" products, MLM promotions. Don't get me wrong, i'm not a cynic, rather, i'm trying to play the part of devil's advocate here. Overall, i'm excited about the idea as I'm quite certain that altruism will prevail and the community will develop well curated website recommendations that make navigating the web a better experience.

check out https://kagi.com/faq

I kind of dig this idea.

Thank you! It's been slow progress because I'm a freelancer in a quiet period, so that's taken precedent. But will start posting progress soon.

I suspect this might be a tongue in cheek reference to digg, a site that does a very similar thing. It was popular for a while before being eaten alive by Reddit. Don’t let that dissuade you, though, there is room to add a new dimension to it or do things differently.

https://en.wikipedia.org/wiki/Digg

this is such a cool idea. i am interested.

Thank you!

do you have anything we can see ?

Still pretty early stages. And I still have some decisions to make regarding the architecture of it. I'd like to do a ShowHN (my first), in the near future.

Reply on news.ycombinator.com