YaCy: a free distributed search engine | Hacker News Reader

YaCy: a free distributed search engine

yacy.net

101 pointse1929300124 comments

Peer-to-peer Information Retrieval is one of those old research topics that face the difficult to solve problem of scalability.

YaCy is yet an early contributor to this research but we've yet to make something to my knowledge that gives comparable results to centralised search engines, is fast and scales well.

Any search engine is good enough, but no one, centralized or decentralized, is even close to Google.

Google is our wonder and our curse.

That's why I use startpage.com, it returns Google results while protecting your privacy :).

I use searx.me for that. It's open-source, and it's a meta-search engine that returns results from more than just Google.

>Any search engine is good enough

That's simply not true. Things we take for granted like true multi-query search and sub-second results speeds simply aren't seen in distributed search engines.

I actually think that the duck (ddg.gg) is better. I now find it difficult to use google, which feels clunky and capricious. I make thousands of queries a day for work, so I really can't excuse poor performance.

For a while I would switch back to google when my searches failed and I wanted to check if the duck was getting me equivalent or better answers. In the past year and a half I've completely stopped. It is simply a high-quality product that exceeds google in what it offers.

Maybe you'll respond that "it just doesn't 'feel' high quality". OK, I can't argue with that because it's your opinion... but as long as we collectively decide to believe this fantasy we will continue to be locked into the search juggernaut. You can help improve the diversity of options by using something different and get to use a better product.

The only thing that I can't do on the duck is search academic papers. Google scholar is still the best thing of its type.

Just another data point but I use ddg exclusively because it's more flexible. Do I get the results I'm looking for? Great. Do I not? I just use to bang syntax to query somewhere else without having to change sites. It's more convenient than google for me to use, therefore better for my needs. I've been using ddg long enough now that I often refer to "searching the internet" as "ducking" rather than "googling" xD but I'm sure that's just because it makes me feel cool, rather than me naturally thinking about ddg before google, haha.

This is exactly my experience. As for the disadvantages of ddg, it's difficult to find some regional results from not-so-popular websites. This is not where ddg shines. Apart from that, I like ddg's results more than google's ones.

I second that! I changed a month ago, and have been pleasantly surprised with the quality of search results. It should be noted that it is possible to adjust the theme so it looks almost exactly like Google, which helped me shake that "unfamiliar" feeling when transitioning. By the way, for academic papers, Microsoft Academic Search seems promising, although it means moving from one juggernaut to another.

The problem is, DDG is not a search engine. It is a UI on top of other search engines.

(Yes, you may say it has DuckDuckBot, but ask around people who run websites if they've ever seen a hit from it.)

I feel like that's equivalent to saying "DDG is not bread, it's a sandwich built on top of other breads"

Maybe Google makes the best bread, but it might be true that someone might make a sandwich on inferior bread that makes for a better overall snack.

In case you don't see the problem inherent in even your analogy: If the bakery goes away, or refuses to sell further bread to the sandwich maker, the maker is fucked.

And there's less than ten bakeries world-wide.

And if the sandwich maker actually got popular enough that people would only buy from them, then the bakeries would indeed give them the finger.

It's not a search engine. But the search engines it is based on provide surprisingly good results. So, I would be careful with the statement that "other search engines are not even close to Google".

I think that's just a matter of habits. Initially, it feels a bit unusual and maybe even uncomfortable to use not the search engine you've got used to. But after some time you understand that now Google is the one that is uncomfortable for you.

Did you reply to the wrong person?

"Your peer cannot accept remote crawls because you need senior or principal peer status for that!"

So the average user cannot contribute to building the crawl index?

That is just the default setting, to prevent using massive bandwidth. You can change the setting to allow for that.

couldn't find it

Sorry, I hadn't used it in a while. You need to open up the port. See here:

http://www.yacy-websearch.net/wiki/index.php/En:FAQ#My_peer_...

This has been posted to HN many times before, this was the most active thread:

https://news.ycombinator.com/item?id=8746883

It looks like there was a big update recently:

http://yacy.net/release_notes/YaCy_Release_1_90.html (Jul 2016)

Wow! I had the same idea over 10 years ago. I'm glad it's being done.

My idea was using a giant P2P indexbase. However, I read the tech page, but it doesn't explain enough. For example, for this to work, there would have to be some kind of "information routing" whereby queries within certain information domains would be routed to machines. I can't see how this is being done.

Because you can't have the whole index on each machine, therefore, how does it know where to route a given query?

Additionally, peer nodes would have to crawl to a certain criteria; either subject based, or perhaps geographically. Otherwise you'd just end up with the same stuff everywhere.

If there's some more explanation, please link.

thanks.

search "learning to learn gradient descent by gradient descent"

The following words are stop-words and had been excluded from the search: [by, to].

Results: 1) Linux-Kernel Archive: By Thread 2) Series Cinematography by 3) Key to Citations

As they are bold, "by" and "to" seem to get boosted!!

So, has anyone used it? If so, how is it?

The last time I looked at this (long, long ago), it was pretty much poisoned by penis-enlargement and drug ads.

I'm hoping it's gotten better since then.

This page took 25GB of RAM on Safari!

Reply on news.ycombinator.com