Hacker News Reader

Comment by dotancohen | Hacker News Reader

Fair point.

How is it to be funded?

is the size of the common Web already way too large to play catch up against google/bing at this point?

my dream, is a distributed/p2p index. each browser contribute to storing part of the overall index, and handle queries coming from other users so that how to fund huge data centers never become a question.

Check out YaCy. It's great if you're happy with slow search!

I had problems with YaCy. Slow search and slow crawling. Regarding search I think it could be improved if instead of HTTP requests to other peers a more efficient protocol (UDP) could be used. Regarding crawling it's quite possible that doing this in Java and running it on a Rock64 may not be the best combination (: It started OOMing after some days.

YaCy's results weren't great and there was the raking was very bad letting sides game it by just spamming a ton of keywords.

  > is the size of the common Web already way too large to play catch up against google/bing at this point?

Probably. But I would prefer a search engine that didn't search the whole web. I would prefer a search engine that searched the sites related to fields that I'm interested in.

So I would pay for or donate to a search engine that provided me good results in e.g. software development. They could add additional fields as demand warrants, so long as quality as maintained. I would even like to see a faceting feature, so I could search for e.g. Matrix and get results on the mathematical concept when need be without having to wade through movie review or fiddle with magic search keywords.

not searching the whole Web makes sense, except that it isn't clear what belongs to your field of interest and what doesn't. should indexing skip a blog page because a the author usually write about algorithm but here goes on and on about business while sporadically mentioning algorithmic technical aspects? and what if you want to search about pottery that Sunday morning, turn to Google? I think segmenting search results by topics is a useful consumer query feature, not sure segmenting what gets indexed would provide a useful service other than covering niches hence not really fulfilling a web search engine. I find the idea less ambitious so maybe that's how an open search engine should approach the problem, federation of hosts could, cover the whole Web eventually.

  > should indexing skip a blog page because a the author usually write about algorithm but here goes on and on about business while sporadically mentioning algorithmic technical aspects?

Yes, because that author's pages wouldn't even be fetched at the point of development that we are discussing.

  > and what if you want to search about pottery that Sunday morning, turn to Google?

Yes, Google still exists. Why not?

why not Google? What I find most beneficial in a search engine alternative is to be able to fetch results or topics I'm less knowledgeable about.

I'm somewhat fine with very commercial search engines when searching for something within a field I'm familiar with, e.g I if search for "kubernetes configmap best practice", skimming infomercial sites, ads, junk is trivial. if I search for "baby milk safety" I have no clue how to digest the results list, I'm totally unfamiliar with the media outlets catering to parenting, nutrition or food health, I know absolutely no authors in the field, no renowned media outlets for tips, not even brand reputations to make a somewhat informed decision on what to skim and what to read with attention.

But I don't disagree an engine focusing on specific fields provides great value. and perhaps that's where it should start to have a chạnce to win.

The plan is to fund it through donations

Can you make it a contributory database? I wouldn't mind "donating" my browsing history and page downloads to build the index and train the algorithm.

You'd have to find a way to verify reputation to make sure no bad actors could contribute.

How does Internet Archive verify dumps submitted by Archive Team and other groups? This may already be a solved problem.

Not knowing their implementation details, I’m guessing it could be doable without reinventing much. An oracle could dispatch a P2P archive job to a pool of clients randomly assigned tasks, with both the first to archive and the first to validate being recognized by the swarm somehow, with periodic re-archiving and re-verification, rate adjusted by popularity of site and of search keywords.

Yes, I'm planning to do something like this.

Reply on news.ycombinator.com