Hacker News Reader

Comment by fdgsdfogijq | Hacker News Reader

Honestly I think people are completely underestimating the difficulty of a good search engine. Google was better ten years ago because search was an easier problem to solve back then. End of story. Nobody is coming along with a better search engine. There is too much spam, content gaming, and money to be made by hacking search.

These posts should almost be blocked from hacker news. ITs a fantasy. Its like saying that democracy has failed so lets replace it, replace with what? Its the best we can get given the alternatives, and its flaws will always be exploited.

View on HN

No. Google is an actively hostile experience. Try it without adblock and get a taste of how most people get treated by Google. And for the search results, Kagi is already better for 90% of my queries.

It's much more difficult now to build a competitive search engine, but saying it's impossible and discussions should be banned is toxic. (And already basically proven wrong with existing competing search engines.)

I think that with current technology, beating google at the scale they run at is impossible. I think with advances it NLP, its possible. But right now, its pie in the sky.

I disagree, Kagi is not yet "better in 90% of the queries" than google, it's good enough to not have to launch google in 90% of it's query.

This is probably true for me, if we also say that 50% of the queries are orders of magnitude higher quality on Kagi.

Kagi's better than Google for me. Proof is that initially I only had it set up as default search on one device but I've progressively switched them all over out of annoyance with crap google results. I haven't yet had to go to google once to finish a search I started on Kagi.

It's substantially better than Google for what I do with it (mostly work-related queries). A few reasons why:

* It blocks all the GitHub and Stack Overflow copycat sites out of the box.

* I can add additional sites to the block list with two clicks.

* It up-ranks (semi-)official documentation like the Python docs or MDN over blog spam sites.

* I can manually up-rank sites (and even pin them to the top of the results) if I'd like to see them more often.

Kagi is awesome because where Google thinks it can guess what you want, Kagi gives you an easy way to tell it what you want.

I already have an extension that's allow me to block, or highlights results on Google or Bing

Came here to mention kagi. Very happy user here. The no-spam results make it so much more useful as a tool. Also, "block this site from myresults forever" gives so much more agency to the user than anything Google has release in years. Googles "we know better" just reads as a big middle finger to me.

Wow, they have that? That’s what I always wanted from Google and they never delivered. Im sure they had reasons. I really don’t think they want to empower their users, they are even hostile to search users and content creators, amp comes to mind and am glad it dissapeared already. Meanwhile I’be been using duckduck but i’ll give kagi a go.

Worse, Google used to allow you to block sites but removed the feature a long while back: https://googlesystem.blogspot.com/2013/03/google-discontinue... .

What happens when you ask it how many Ukrainian generals have died? Does it insist on giving you pages and pages about Russian generals, like Google?

I was looking for this and it really annoys me how it thinks it knows what I want.

No, Kagi (unlike Google and DDG) will just stop providing results, it won't make up others with very tenuous links to your search terms like the others do.

Just tried that. I did have to put "ukrainian generals" into quotes (which makes sense, that's how it's supposed to work) but then it just showed me 5 results total, all of which contained exactly that phrase.

How are the porn results?

+1 kagi was the first alternative engine that has passed all of my random tests, mostly weird cameras and python coding searches…

But why would a search engine require an account to use in the first place? Or maybe I took it wrong?

Kagi requires users to sign up (even though it's currently free?) which is 1000x more user-hostile than anything Google does and makes it a nonstarter as far as I'm concerned.

Kagi is intended to be a paid service when it launches. This is something I actively want. It should make them the opposite of user-hostile. Their users will be the source of their revenue so they will need to provide value or lose them. Login is a necessary part of that. I'm happy to take both that inconvenience and the cost.

I used it for awhile and did like it quite a bit - I found myself having a bit of anxiety though worrying if I was going to hit the search limit with my more trivial queries. Google really has us psychologically

Is there a search limit for Kagi during beta? I don't see it documented anywhere, and I've been using it as my primary search engine for a few months and haven't seen any sign of a limit.

I wouldn't be surprised if there's some limit to prevent abuse, but it hardly gets in the way of my normal usage.

According to a discussion I had here previously with a Kagi user, when they go out of beta they plan to offer a $10/month plan with a quota of 20 searches per day (and $0.015 per each additional search), and an unlimited plan for $20-30/month. Additionally, every time you change something related to your search query (such as clicking "Images", changing the sort order, applying filters, or blocking a website) counts as an additional search, so you could go through your quota pretty fast.

Apparently all of this is to offset the costs from paying Google and Bing to run searches for them (the site basically takes results from Google and Bing and then combines them together with their own special sauce) but with a pricing model like this it seems like Kagi will remain a niche tool solely used by the wealthy.

Hm, 30 USD/month for good search is not that much - and certainly not "wealthy" territory. Remember that many people pay subscription services they use a lot less (18 USD for Netflix, anyone?).

The main paradigm shift necessary will be to pay at all for something that was "free" and is provided "free" to this day - even if the quality is worse.

Monthly subscription and quotas? While I respect and understand the goal I can’t help but feel like we are paywalling what the web has been for the longest time and should be “out of the box”.

And as of now, they are listening to users very attentively. Just a few hours ago I suggested they clarify the defualt "Programming" filter since it only searches q&a sites (i.e Stack Overflow) and they have already changed the name to "Programming Help" to make it more clear.

Hostile is such a drastic word to apply to a service that is dedicated to making search better by actually listening to the desires of their users.

They need to support their efforts somehow. Instead of showing ads or selling your data, they are going to charge a fee to use it.

Google uses ads to pay for the service, which is available without logging in. Those ad systems can track you. On the other hand, a paid service (even if it is currently in "free" mode) will likely get rid of ads, but requires some authentication to know you are you.

I'm logged into Google all the time. I do this to get sharing working across Chrome. Others doe the same with Firefox. Some don't, because privacy.

I hear you think logins are 1000x more hostile than paying with your eyeballs. How would you suggest resolving this clearly conflicted view with logging in vs. how the revenue is made on a given site?

It used to be that if you used a paid service, then that service wouldn't throw you to the wolves of internet marketing/tracking companies. That alone made logins 1000 times less hostile.

But that is changing. Internet marketing companies are now teaming up with websites so the websites to their dirty work for them and you end up getting tracked anyway -- but with much higher certainty as to who you are. That means that now, yes, logins are more hostile.

The web continues to get smaller and smaller.

And they also intend to charge $10 per month (and that's just for 20 searches per day, which is a WTF low number). There's no way I'm paying that kind of money.

Even my grandparents figured out how to scroll by the ads at the top of Google search. Actively hostile is a bit much.

I don't think my parents would be able to distinguish between the ads and natural listings.

I use Google without AdBlock on my phone daily and it's not quite that different.

I use Google without AdBlock whenever I am in a private browser in addition to going without adblock on both my phone and tablet. I agree, it isn't that much different. At least with search, that is.

Kagi is probably a lot more usable, right now. Nobody’s trying to game it.

As long as it remains niche, it might actually stay that way indefinitely. I’m just worried about Google taking inspiration from them. I bet whatever “tricks” they use now would get a lot less effective if they did.

Do I have to wait for their invitation to join the beta list before I can try Kagi? The comments here are promising and I tried to take a look for myself but apparently only beta users are allowed in at the moment.

You have to ask for it by submitting a form in their webpage and they send you an invitation to your mail

What’s the story behind Kagi - are they somehow related to Google?

To my knowledge besides the old experiments with client side search, Google has never allowed (even paid) api access to their search. I thought they did not do this, because it would have allowed somebody to jump start a search engine that might eventually become a competitor.

Kagi seems to be open about their Google relationship, so I assume they have agreement in place.

This even feels like some experiment from Google to create a premium, paid, search product (like what they did with YouTube).

At least Startpage has been serving Google results (via paid API access) for years. There is nothing new about Kagi paying for access to it.

The progression that people almost always forget:

1) New system comes out that indexes/controls/regulates a naively created dataset

2) Data consumers adopt that system and experience benefits

3) Data suppliers learn rules of system and take advantage of it to improve the positioning of their data, thus breaking the intent of a system built on the assumption of naive creation

4) Users complain about the broken system

5) New entrants realize that the original system actually solved the core problem really well, and there are no easy ways to solve the 'gaming the system' problem

6) Flawed system remains the best available option indefinitely

It's like entropy, there's just no fighting it.

EDIT: And to extend this beyond Google - do you see a lot of long text blocks in 7 second tik tok videos? That's because the creator found a way to game the algorithm.

3) ban abusive data providers?

How would that even work? Define the border between SEO and abuse.

> Define the border between SEO and abuse.

Perhaps there is no border to be defined since nearly all SEO is abuse?!

The only acceptable SEO should be "provide website content that is more relevant/interesting for the viewer".

Let's say someone writes a blog article. They publish it on Wordpress with a bunch of unoptimised defaults. Some time after, someone comes along and says "Hey, the search engine will rank your post more highly if you include the name of the post in the URL instead of just having it be blog.example.com/posts?id=1234", so they change the URL. Is that SEO? Should that result in the site being de-listed/banned? It doesn't meet your definition of "provide website content that is more relevant/interesting for the viewer".

And where is the line between "provide website content that is more relevant/interesting for the viewer" and SEO? If I realise that (to take an example from something found on Google) I could take the sentence "Identify the best customers and convert more" and rewrite it as "Marketing automation helps you identify the best customers and convert more", is that SEO because I'm intentionally adding a keyword, or is that making the content more relevant to the user by being more explicit in what I'm saying?

To my moral standard, all of this is manipulative if the intention is to manipulate the Google ranking instead of seeing an improved Google ranking as a result of delivering better service to the visitor.

This does of course not imply that these things can be proven, which implies that no action can be taken. Not everything that is immoral is punishable.

I'm sure there are better examples, but this specific one actually does make the experience better for the reader. People often return to good blog posts and it's a lot easier to remember a title than a numerical id. If you can remember the title and know the url of the blog and it has a simple naming convention, you can bypass the search engine entirely and just go straight there. Sports sites are really good about this. <URL>/<LEAGUE>/stats always gets you the stats. <URL>/<LEAGUE>/standings gets you the standings. If you're curious how a team is doing or who currently leads in scoring or whatever, you know where to go without needing assistance from search.

Well let's make this real.

Sites with people hired to optimize the robots.txt file are ranked higher than those without. If we ban all websites that hire people to optimize their robots.txt, then a lot of the web isn't indexed and the search results suffer.

Or if we ban every creator on tik tok who posts a lot of text in a short video, then a lot of creators are going to be de-platformed and moved elsewhere.

The point of what I wrote is that 'humans are really god at following rules, so much so that they often are able to manipulate rules to their advantage, without breaking said rules.

That's why this is such a hard problem to counter.

You're ignoring the only insightful part of the article: if a search engine succeeds at all, it cannot ban anyone, because some antitrust bureaucrat somewhere will cluck about it.

If this were true, wouldn't Google display all million pages that are vaguely relevant to my query on the first screen?

Their only job is to stack order the internet. Down ranking a site to the 1,000,000th result is the same as banning them. They necessarily do this all the time (like for every search with more than 10 potential hits).

Honest question: what else besides web search has followed this progression?

Spam and bot detection could be close to that model ?

Email spam? I'd say that's basically solved.

For bot detection, when was the golden age? I'd say that, rather than follow the above description, it started as non-existent and there was basically a smooth descent downward to where we are now.

I believe blocking an honest discussion is never going to do more good than harm. The original post doesn't even suggest that we should overthrow our Google overlord to replace it with whatever we find interesting. It's just pointing various limitations and issues that Google users are facing right now. And apparently from the replies here, there are even more aspects that people are have troubles with.

It's true building a good search engine is an gigantic undertaking but if Google (or someone else entirely) is aware of the current issues, they may have ideas to tackle those problems and then we can all share a better internet.

Same for the argument with democracy and any other similar arguments, pointing out problems within our current institutions doesn't remotely mean that we want to abolish everything and start from scratch.

> Nobody is coming along with a better search engine

https://neeva.com/ better than pretty much any solution (Qwant, DDG, etc) I've personally tested. It also indexes specific websites like StackOverflow, GitHub, and GMail.

Edit:

Neeva does require an account to create because eventually the product is going to require a subscription.

"See results for '<my search term>' Create your free Neeva account."

"To continue searching and access all of Neeva's features, create your free account. Already a Neeva member? Sign in"

Yeah, no.

Ah, Neeva is eventually going to be a paid product, that's why it's that way. That said, their membership is pretty cheap for what you get.

I don't mind giving my email or making an account after I have any inkling that the product might be worth deleting a few spam emails. Neeva was giving me a modal popup in my first minute; that's never going to get my email.

(Founder of Neeva here) Appreciate the input and hear your frustration. This is a carryover from changes we made to introduce the free tier earlier in the quarter. We are making imminent improvements on modal frequency and getting them out of your way. Stay posted.

Yo! Neeva is a dope product, imo, but could I offer some suggestions on how I use Neeva?

1. I have more than one GitHub account (one is for work, one is for personal stuff). I would like to somehow keep them separate, but at least being able to add multiple GitHub accounts for indexing would be cool.

2. Could Neeva offer something for segmenting the parts of my online presence? For instance, there's personal me and then there's work me. I'd like to turn a different profile on when I'm doing different things.

3. I don't know if it's possible to partner with ProtonMail, but I'd love to index that and my Proton Calendar as well.

Also didn't know the free tier is permanent, but I think that's probably a good move. Thanks for what you do!

Hi! Thanks for your notes here. I'll take these suggestions back to our team. re: 2. Being able to connect two accounts is interesting -- right now I solve this by having two Neeva accounts, one associated with my work email and one with my personal email.

I get that.

But essentially we all have de facto accounts on Google anyway.

They track and monitor usage and then map you to a user account when you finally login somewhere.

I'm totally fine with that being upfront and made clear from the get go!

I experimented with neeva, because I'd really like an alternative to Google, even a paid one. However, I found its results pretty disappointing. At least for my work, I had to go back to Google, because I don't want to waste time "on the clock."

DuckDuckGo has been good enough for my day-to-day use. I’ll use Google for a few things.

Neeva has a free tier (with only access to the search engine, so people can still test the search aspect), but when you pay for it now you ALSO get LastPass premium and Bitdefender VPN Premium as well, along with all their integrations into your system.

If you trust these people are doing what they say, it's a pretty good deal for securing your internet spaces and trying to get away from google, imo.

Not sure if you read the post. They don't advocate for replacing Google. They want to add onto Google and other search engines.

Sorry but duckduckgo or probably any other search is better than Google at this moment

Not everyone lives in the US and/or an English-speaking country.

I can't find shit with DDG and the experience is like using Google from 20 years ago when it didn't have any of its bells and whistles.

It’s actually because I’m not in an english speaking country that Google is failing me and screwing my searches for anything than can be somewhat pushed to local results…

I’m glad it works for you, it just reminds me how different are everyone’s expectations of good search results.

Same here. Traveling and when I search for postgres I don't want to find the local student project as the top result

Some side by side comparisons just now:

"pizza near me" - Google suggested a well rated place within 5 miles. DuckDuckGo suggested a pizza place 91 miles away??

"busted kids lip" - Google says gauze and a cold pack. DuckDuckGo says salt water??

"best monitor for mac" - Google says Dell Ultrasharp with 4.5 stars and 738 ratings on amazon. DuckDuckGo says BenQ 4 stars with 174 ratings on amazon?

how is this better?

> Honestly I think people are completely underestimating the difficulty of a good search engine.

No, they are underestimating the difficulty of funding a good search engine.

I liked the runnaroo search very much as did several of my friends. The guy who ran it couldn't fund it. He shut it down.

Altavista (Yeah, that far back) had a nice feature where it would draw a cluster graph of your search results. So, if you searched for "python", it would show your results but would also draw a little graph and you could see that "Hey, there are two clusters here--programming and reptiles." You could then click on the "programming" node and the "reptiles" cluster would go away. It allowed you to drill through irrelevant stuff really quickly.

Note how that feature doesn't exist today--in spite of orders of magnitude more programmers being thrown at search, graph algorithms, and nifty Javascript web UIs. I wonder why ...

(/sarcasm in case you missed it. I don't wonder why. Such a feature would let you drill through irrelevant Ad and SEO garbage too quickly and would impact Google's revenue.)

building Breeze, which does pretty much that minus the cool UI angle, and yeah, only way for it to work at scale is premium like neeva, kagi, etc.

we started out with topics and then moved to web, and are now folding topics into the web search experience, it's really hard stuff to get right

our first main filter was blogs, which is getting renamed to posts for a mix of reasons, adding forums shortly, along with other more specific topics

re: https://breezethat.com/ -- & better on laptop / desktop atm, premium version will be ad-free

I'd say you're half right. Things are harder now and the success of Google has contributed to this.

However, I think Google has severely degenerated from just two years ago, when most of the problems were fully in effect.

Google is a bit of a product of the situation of scams being the easiest way anyone makes online.

> Google was better ten years ago because search was an easier problem to solve back then

Has anyone considered the possibility that all this Machine learning and AI models is what made Google and YouTube searches worse?

I strongly suspect that's a very large part of what is making everything worse. Or, at least, there does seem to be a correlation in time between the implementation of ML and the degradation of the quality of results.

I wouldn't call it so much the implementation of ML but rather from when they decided they were an AI/ML company.

The Amazon recommendations engine, the poster child of ML in 2014 isn't any good at recommending products that I actually want to buy either.

People said this about pre-Google search engines. Someone will figure out the next PageRank and give us another 10-20 years of useful search.

There were many multiples less people online. The internet economy was almost nothing back then.

There’s no comparing how things were in the 90s and early 00s to now.

Every social media company before Facebook faltered or began losing a good chunk of their user base fairly quickly. Until that stopped happening with Facebook and IG for over a decade now.

Reddit’s position has also been here for a decade now.

I don't think these posts should be blocked, because they create discussion, which creates interest, which incentivizes for problem solving.

Humans are incredibly good at solving engineering problems they can see from a mile away, although it takes time to solve.

Could you please go into more detail , I agree with you btw.

  Honestly I think people are completely underestimating 
  the difficulty of a good search engine

I suspect that is not difficult so much as expensive.

Boutique search engines pop up all the time here on HN, but they can't compete fairly against Google, without the resources to crawl a billion webpages day after day.

Google also keeps a copy of the "entire" web in RAM to search it faster.

Source? But it's definitely possible nowadays when RAM is cheap and modern Google makes the Internet feel incredibly smaller than it was 20 years ago.

At the time you would get tons of results from a myriad of small blogs, forums, niche websites. Nowadays it's Pinterest, blogspam and more SEO optimised algorithmic crap. If you're lucky you get a forum result that might actually be relevant.

Given the joke that if you can't Google it it doesn't exist, it follows that the Internet has considerably shrunk in the past two decades.

It looks like I was mistaken. They don't keep all of the documents in RAM, but I'm pretty sure they keep the search index (100 petabytes) in RAM. I'm struggling to find a source but I remember hearing it in a talk given by a Googler on YouTube. I also read somewhere that moving their whole index to RAM is how they brought their response times down from a couple of seconds to a couple hundred milliseconds.

Here, let Jeff Dean explain how it works (and why this is a complex problem that a lot of the naive 'build a better search engine than Google' miss)

https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8...

The original article's description of how Google search signals work doesn't even scratch the surface of all the signals it's using.

A link to the talk for those curious

https://www.youtube.com/watch?v=IQuSuyUVkoE

Democracy would be best replaced with cashless society. Society based on everyone value of doing what they love doing most (aka hobbyist), and virtually everything else replaced by robots and technology we already have. This is what's coming eventually, but its not something that can be installed on the top of your "operating system"; you will have to format the whole hard-drive (civil war)

I suppose in the absence of cash we would just force some people to do the necessary tasks that aren't anyone's favourite hobbies?

Well, ideally people could choose to do whatever they liked to provide for society. Everyone would have a role to play. So those that like to farm (raised on a farm, to be proud of their work) would farm, those that want to commit their life to science could attempt to advance their field. Someone might find an invigorating pride in making sure wastewater treatment is running well, but they don't know how to get into the field and are stuck working a job they don't like for money, currently. Artists could create culture. As long as you are doing something to provide for society, society will provide for you.

There will be people that do not have favorite hobbies. There will be people with no ambition who benefit from being told what they can or should do to contribute. Those people, in order to have their needs met could choose to take on a Necessary Task, even if it isn't their favorite thing. Maybe they learn to love it and excel at it.

Don't want to do anything to contribute to society despite being physically and mentally able? No needs met. Meet them yourself, you magnificent lone wolf. Want your needs met but don't want to do Necessary Task that you hate? Find a purpose and pursue it.

Couldn't have written it better myself. Thank YOU!

I don't know how to break it to you, but the system you're looking for that does all the things you're saying is capitalism.

Have the talent and desire to be a heart surgeon? Go for it, and you'll be rewarded. Want to make life sized statues of the Beatles out of used printer cartridges? Eh, fine but maybe also get a day job.

Ah yea. Let me just pay to put myself through med school independently while I struggle to pay for food and housing.

Sorry but Capitalism is a broken system that is not meeting mine or million's of others needs, much less enabling us to pursue what we're actually passionate about. I at least know that I am passionate about building and supporting technology but I still struggle to pursue it professionally because I had the poor luck to be born without the resources to get a piece of paper that says I know how.

It's sort of weird how badly people have become unable to understand what the word capitalism actually means, these days

It seems like everyone just thinks it's how you describe the contemporary world

No, what they're describing isn't capitalism. Not even close

I don't understand how any of that has anything to do with democracy.

Reply on news.ycombinator.com