Hacker News Reader

Comment by xnx | Hacker News Reader

As the article mentions, the web has changed. It's very hard to return a list of good results when there are none. I've yet to see side-by-side comparisons where another search engine is doing a better job.

I've had "Search Generative Experience" (AI/LLM synthesized result/summary above the standard web results) turned on for awhile. It is definitely the future. To survive the flood of revenue optimized content (recipes being a familiar example), you need an active [software] agent working on your behalf to read content and make one straight thing out of the crooked timber of the web.

View on HN

The problem isn't just that the internet has become more SEO-oriented and encumbered with low-value clickbait websites -- it's also that Google doesn't listen to instructions the way it used to.

Google started slipping when it began assuming that it knows better than you do. For instance, when the exact search operator (" ") stopped working.

I'd take the Google of 2014 over the Google of 2023 without thinking twice, and I think that I'm not alone in this.

Verbatim search still exists.

https://www.google.com/advanced_search

Do you use it often? It doesn't work the way it used to. I can't count the number of times I've searched for a technical term -- in quotes -- and got results back for a subtly (sometimes not so subtly) different term.

you have an example of this? using quotes is not the same as going to advanced search and saying use exact word or phrase. silly, yes I know.

Quotes is different than verbatim mode. Compare:

[verbatim mode] https://www.google.com/search?q=unexpected+newline+in+compos...

[quotes] https://www.google.com/search?q=%22unexpected+newline+in+com...

The top two results are the same, but besides that verbatim mode seems way better to me.

I don't even bother with quotes anymore. When I'm searching for something technical from the normal interface, I'm almost always going to Tools -> All results dropdown -> Verbatim.

I see that I can specifically ask for results that are in the Flash .swf format. Very convenient.

Agree. Also their results are often biased and I sometimes find myself going to Yandex.

I think Frank Zappa would chortle with irony that a Russian search engine returns his discographies uncensored in the here and now versus an American one.

All search engines are inherently biased, since they perform a ranking task (which is a value judgement). They all encode (possibly uncessfully) the values of their creator. This is why having multiple search engines is important, as diversity in search options is the closes you'll get to objectivity.

(said the independent search engine creator with the same air of righteousness as when Plato argued philosophers should be kings)

Yes but they’re also politically biased. Not to mention, the enormous amount of web content that is effectively banned from seArch engine results due to low prestige

Congrats you are one step from re-inventing DogPile. Add AI and blockchain to pull in a ton of VC money.

Absolutely. i so often do a search and it presumes to think it knows what i want rather than giving me what I ask for, sometimes even when I'm being quite specific. I will say it does a great job for programming stuff but a lousy job whenever I'm researching economics topics.

Even programming stuff it does a lousy job. I use R a lot. I plot with the base graphics package because it works fine. Finding R information that does not involve the tidyverse library is so damn difficult on google search. Its bests to just go straight to the documentation since stackoverflow and other sites are pretty worthless with pretty much all answers requiring ggplot or dpylr even if they can be implemented with zero dependencies with the base tooling.

Same situation with python honestly. No I don’t want to use pandas, if I wanted better data wrangling I’d just wrangle in base R then import the wrangled data into python if need be instead of adding another dependency to my python code. Apparently this makes me a heretic based on google search results.

More of the web is also behind paywalls or is on platforms like tiktok, instagram, twitter, etc. Less to scrape and index.

My most recent complaint is the disappearance of the pagination numbers.

Since I know the top results are mostly useless spams on some searches, I used to go to page 4 or 5 to find the actual results. Now to achieve the same result I have to click and scroll 5 times.

>For instance, when the exact search operator (" ") stopped working.

It still works.

It works, but it does something different than it used to. It used to only include results that actually contained the exact phrase you put within quotes. That is no longer the case.

Now you just check "verbatim search" in the tools menu.

Works exactly how quotes used to.

It works the same and it never only contained the exact phrase as Google ignores punctuation.

This is not my experience. I search for exact phrases the same way I always have. Three to result does not contain the word I emphasized. This never happened before a few years ago. Now it happens daily.

Are you sure the page doesn't contain it? It is not a simple as just visiting the page and searching for the thing you quoted as there may be punctuation between words or the text may be in the alt text of an image or somewhere else that is not obvious.

I am quite sure. The kinds of words in question would not even really make sense for "SEO gaming" to come into play. Typically, it presents with trying to match a word vs a phrase(I don't ever try to match phrases). I try matching smaller and smaller fragments of the word and still... nothing. I remember this working back when it was common for pages on gray websites to stuff giant blobs of text with juicy keywords out of view with janky html. Yet, somehow those pages were ranked very low and results were good. There is a clear degradation in quality.

Actually, it has happened with results that include hacker news discussions. I scrape through and expand EVERY comment on a 6 month old post and still find no partial or exact matches for my search keyword. It is very clear to me that Google has switched behavior. It used to flex on how awesome it was to bring you to an esoteric corner of the internet matching your exact query. Now, it tries to converge your queries into what it thinks you want(not what you asked for) and funnel you into large websites at the expense of utility.

It will substitute similar things, such as the word ubuntu for debian.

Which can be incredibly frustrating if you know the problem well enough to need a different answer.

This is a good point. I searched for "rename" today and the top result switched it to "name" instead. The TOP result completely changed the query. Infuriating.

I also love when my search is narrow enough that Google instead starts to return more generic pages for totally different programming languages instead of the smaller result set that matches exactly. It is very easy to observe as you explore a topic. At some point, you got a wall where any increase in search specificity seems to yield worse results than queries that helped you get to that point.

Yeah; that's a good point. Google's desire to provide popular/well known information sources is outweighing the ability for people to be specific. It's infuriating to be ignored as a user. Hence using verbatim constantly, which feels like a checkbox for "stop failing me".

Which... Why isn't Google learning that some users are constantly seeking out verbatim for results they'll click on and adjusting their search results to be more verbatim?

I'll also had that a huge portion of genuinely good content is now inaccessible to Google in walled gardens like Discord, Facebook, Twitter, and TikTok.

I remember google images before Getty and Pinterest ruined it.

So, how paid search engines like Kagi manages to bring useful results every time?

Its results are superior, with way less spam and SEO optimized content.

I have no idea; but I speculate this is one area challengers have it easier because

a) there is no giant industry optimizing for ranking on Kagi. There are whole armies of people trying to game google search, meaning whatever heuristics they have would probably be rendered useless shortly if google were to adopt them.

And

b) Kagi has a much smaller, more technical userbase. So like the early web they can get away with a search experience tailored for those kinds of people and not millions of people powering on a smartphone as their first computer.

The biggest power of Kagi is “you” are the algorithm. A page is optimizing for Kagi and providing bad results? Block them. You found a good website burrowed down? Raise/promote them.

Kagi works for you, with your input. It doesn’t do “advanced guessing” by implicitly and opaquely processing your interaction with it.

Explicit control can work for ranking but can’t scale to the retrieval step of search - you aren’t going to sift through thousands of thousands of pages of potential results to give feedback on a meaningful portion of them, there must be some method of pulling pages to be ranked. This is where the gaming happens, with basically limitless spam domains that need to be filtered out. There just is no way Kagi is 100% relying on self reports for this.

The whole idea of the model is to fit it on some reference so you can project it onto new data you haven't classified yet. So your first few weeks, you might get some spam sure. But eventually if you are optimizing the system by e.g. only ranking technical blogs with no seo bullshit, then technical blogs with no seo bullshit is what would fit best with that model among the new data and what you see on your results. That can't be gamed without knowing the sort of model fit you are generating and this fit is defined by the user.

The ranking is only relevant to you. It's part of Kagi's lenses and search tailored for you, by you approach.

I guess their "Universal Summarizer" is a part of their indexer, so they try to understand what you want and provide you more relevant results.

They probably have other signals and systems to detect spam and other unwanted things.

Also Kagi is still in the phase of "let's give users what they want" instead of in the phase of "let's give shareholders what they want".

Since users are the ones who're financing Kagi, that phase will have no end until they become free and sell user data in a way.

A lot of people on HN are very happy with Kagi, but I haven't seen examples of searches to understand where it is better. In a blind "taste test", I would be surprised if Kagi or Bing is preferred to Google for most searches.

Well, all my daily searches, whether it be technical or daily mundane things return better results with less cruft than Google. Also, it notifies you if a page has excessive number of trackers.

Combine it with personal website ranking and lenses, it surpasses Google easily.

Also since it has no ads, there’s no noise.

I don’t use Microsoft products, so I can’t comment on Bing.

I did side by side for a while and Kagi was consistently superior to Google and bing. I think it boils down to they don’t try to cross sell advertisements so they’re more oriented towards actually providing better results rather than better paying results.

Kagi's killer feature can't be blind tested, because Google and Bing don't have it. The ability to pin, raise or block sites in your results is a game changer.

Interestingly, Google had that in 2008 (SearchWiki), but it got extremely low usage - enough that at a time when Google was serving 30K QPS, you could read the queries with SearchWiki interaction as they came in on a screen.

I suspect this is a niche feature for power-users only. HN is filled with power users, so they love Kagi, but it doesn't necessarily translate to the masses.

2008 didn't have Pinterest, fewer listicles, conspiracy websites were less mainstream, and mobile web was in its toddler phase.

The feature was basically used exclusively to kill ExpertsExchange, so the problem existed, but yes, it wasn't quite as bad as today.

Well, you would still compare Google as-is. It’s an apples to oranges comparison but you can still compare apples and oranges side by side.

I think it’s important to measure and quantify these things.

"How do these products compare if I ignore their unique advantages" is not a useful measure.

Do you need examples of searches? If people get more useful results from one search engine over another, that's all that matters.

I can't provide you with side-by-side examples because I don't compile lists of such examples. I'm searching to find stuff, not researching search quality. I just know that I have more problems getting good search results from Google over a couple of the other engines.

Yes. I would love to see some actual examples. People have said that verbatim search doesn't work for them and I've never observed this, so I'm also skeptical of claims of dramatically better search results.

So you think everyone is lying?

No. What they consider "better", I might consider "worse".

I can almost guarantee you that you'd find better results if you could block all results from spammy domains (without using an extension). It's practically a tautology.

It's probably a good thing that just one week ago Google rolled out their "helpful content update" that incorporates a site-wide quality signal: https://developers.google.com/search/updates/helpful-content...

It looks like this uses some sort of automated system controlled by Google -- I am skeptical we'll see any improvements, given how bad results have been for so long (see all the SO and GH clones returned by Google for many examples). Who knows, though; I've certainly been wrong before.

No one can provide examples. But it doesn't mean they are lying. They may think it's better but actually isn't. Maybe it is worse but people are protective over it because their identity is connected to it's success.

> They may think it's better but actually isn't.

If a user of service thinks a different service is better, then it is better for them by definition. "Better" is a subjective thing. If X seems better for me than Y, then it is.

I cited [above](https://news.ycombinator.com/item?id=37618517) a Google search for

"Are US drivers licenses forgery-proof?" and "Can US Real-ID drivers licenses be forged?"

gave utterly irrelevant #1 or #2 hits: nothing to do with US or Real-ID (zero mentions), it's about New South Wales, Australia (lots of mentions of "NSW"). (although from US magazine Wired.com, but Google can still tell this article is totally irrelevant).

2) I previously in 2022 posted "Google search relevance fail: result for “Africa longitude” [https://news.ycombinator.com/item?id=30337563] where the #1 site hit and text result were the grossly misleading "Africa/Coordinates 8.7832° S, 34.5085° E", which is not a range, or even a centroid, just a junk location in southern Tanzania, where a (Russian?) SEO spam person seemed to have dropped a bogus pin, mislabeled "оф. 3, 27 ул. Хабаровская, Bila Tserkva, Russia". In Tanzania. 17 months later and same gloriously unfixed garbage results.

Other bizarre top hits were the small Ottawa e-commerce digital-map site https://www.mapsofworld.com/lat_long/africa.html, with the incomplete and inaccurate non-sentence "Africa's latitude and longitude lies between 9.1021° N, 18.2812° E." (That e-ecommerce store seems to have an autogenerated page with incorrect information on latitude and missing longitude for each country).

These are cases where Google itself 5 years ago would have done better. Or compare to Kagi.

I’ve thought about making an account and spending the free quota on a side by side comparison. The problem is you need a representative sample of queries. The things I search might not be what you search or what my mom searches.

80% good enough results and this [1] was all I needed to switch and not look back.

[1] https://kagi.com/privacy

"To survive the flood of revenue optimized content (recipes being a familiar example), you need an active [software] agent working on your behalf to read content and make one straight thing out of the crooked timber of the web."

So what do you think, will be the goal of all advertisers?

To get their recommendations baked into that AI. Direct bribing/revenue sharing with the manufactur of that AI. Or sneaking data into training sets via bribed employes and so on.

I don't trust black boxes and never will.

Perhaps open-source AI models will be the our counterweight to commercial capture of AIs.

Then these companies will just use closed source models.

Right. That's why the agent should live under your control, and receive training from your search history.

Paid search is the future. Either you pay a subscription fee to something like Kagi, or you pay for hardware and electricity turn run your own "AI agent" locally.

I don't think the problem, in general, is that no good results exist for any given query, more often the problem is that the good results are rarely well search engine optimized, and thus stand zero chance of making the search results.

Some people admittedly expect pretty weird websites to exist. Like who would even publish a good objective comparison between products in a specific category. But people make those queries looking for those types of results, and spammers generate such content to make bucks off the traffic. 20 years ago you'd have gotten nothing in most cases.

There are a lot of people that genuinely make comparisons of specialty products.

That sort of stuff is typically marketing.

Or simply just passionate people talking about a niche.

Consider coffee. You'll find hundreds of articles comparing the most high-end grinders. You even have PhDs writing white papers on the physics of coffee and how particle size distribution achieved by different grinders affects extraction quality. Those are genuine opinions from aficionados.

The same is true for all kinds of other things. The internet has a lot more than the auto-generated marketing crap you give it credit for.

This is true for things that are related to a hobby. There are many things where it basically doesn't exist. Nobody is writing articles comparing server colocation offerings near ${YOUR_CITY}, yet people search for them expecting them to exist.

That's not true, there are plenty of articles discussing these sorts of topics written by actual people.

Sniper in mahwah for example will discuss some of this for example from an HFT angle.

Such people exist in every city around the world?

Finance markets are spread throughout the world and people spend a fair amount of money building efficient networks to route data between them faster than the competition, yes.

Dude, there are fewer markets with a flash boys scene than there are cities in Kansas.

> To survive the flood of revenue optimized content..., you need an active [software] agent working on your behalf

How long do you think it will be before that software agent's output will be just as riddled with commercialized content? I think we might have only a short window where we get to enjoy the fairly unadulterated output of the base LLMs by the large providers (yes, I'm aware that they're screening for safety already, but there could be a lot more coming). Imagine a dialogue like this: Q: What should I watch out for when buying new car tires?

A current answer might be something like: Here's a list of things to watch out for when buying car tires: A, B, C, D, E. Remember to always consult with specialists. Shall I explain more about the topic?

A future answer might look something like: According to experts like the professionals at [tire shop near you], here are the things you need to watch out for when buying car tires: A, B, C. You can use this coupon to get a 10% discount off your next purchase at [tire shop near you]. Shall I book an appointment for a free consultation at [tire shop near you] for you?

I don't believe that anything Google does now - summarization included - will avoid being gamed just as effectively as their current search results have.

It's a negative-sum arms race, but if Google (or whoever) wasn't spending the money to fuel the arms race by passing on ad money, who would pay for Google to exist in the first place?

> you need an active [software] agent working on your behalf

I'd love to have one, but do any exist? SGE works on google's behalf, collecting data for google who will use what they learn from it against me. I'm guessing that until we can use common hardware to train/host an AI agent ourselves, we're stuck with terrible search results unless we're willing to hand over more and more data to others who want to exploit us, while pretending we can trust them to return results that are in our best interest instead of theirs. That's not a great future to look forward to.

I am convinced that a lot of the reason why Google search went downhill is that they started using more AI. Specifically, using it to try to interpret what I'm searching for rather than taking my word for it, which leads to irrelevant search results.

That's just my hypothesis, of course. I don't know for certain. But it does make me very skeptical about using AI to help search the web.

I want a robust set of search modifiers instead.

I liked StartPage/Ixquick results much better than what Google used to show. Maybe they still do but wouldn't know as I stopped using it after they got acquired. Didn't/don't they rely on Google?

Reply on news.ycombinator.com