Ask HN: How do I search the web in the age of enshittified search engines? | Hacker News Reader

Ask HN: How do I search the web in the age of enshittified search engines?

51 pointsManlyBread36 comments

Like most internet users I used to use Google to search for information but now the quality of the results is abysmal and I no longer find it useful most of the time.

I've tried other search engines (Bing, Yandex) and the results are a bit better but still nowhere near the old Google. Kagi is a thing but I'm not sure if the quality justifies $120/year, especially since I don't earn my salary in USD.

Putting "site:reddit.com" helps a bit but there's plenty of stuff that simply isn't there, so it's not a solution either.

LLMs are a bit better with the quality of the searches but hallucinations are a thing so I need to verify the information myself so I am back to square one.

Is there anything else I can do to find information easier or to improve the quality of the searches?

View on HN

If you know the below mentioned, then just ignore. Otherwise give it a try.

There is one thing, that many people either forgot, or never knew: how to create a search query. Nowadays most people put in a human question: "how to bake bread", "how to use a red toilet seat", "what happens to today around the corner".

However even to this day, search engines gives better results if instead this, you try to imagine the results, and search for text that you think appears on the correct result: "bread recipe", "toilet seat user manual", "concert Tuvalu 2024 november"

The HYDE approach as it’s called in RAG vernacular.

This. Search engines are an index, not an oracle. Even though google and the others have tried to mystified it by adding those answer boxes and doing a bunch of natural language processing. Another trick is to use the `insite:` keyword. Works wonder for searching docs.

That is how I always did Google searches, even from 20 years ago, and I still get garbage results.

Well, yeah. I didn't say that it will improve search results if you are already doing it like that. I do agree that google gives garbage since like 2020, but you know... with natural language it gives steaming crap. With real search queries it gives only crap.

I no longer find this useful because I tend to get the same results no matter how I format my query. And these aren't even good results. Most often I get results for the stuff that is similar but at the same time completely unrelated to what I am actually searching for.

An example from yesterday: I wanted to search for curtains that would be resistant to cat scratches. No matter what I'd search for I would get the same set of links including the following:

* shops that sell textiles that are cat resistant but aren't drapes

* various surface-level guides that talk about this subject but no mention of any specific products (SEO spam)

* ways that are """helpful""" but ultimately harmful, such as spraying smells that cats doesn't like

Not a single mention of anything actually useful or related to what I actually searched for. I am either the first person who asked about this topic ever or there's something wrong with the search.

I'd say "-site:reddit.com" helps a lot. And also "-site:youtube.com" and some others :)

Admittedly, I usually do the exclusion in my brain and not actually via the search query.

I don't know where the hype comes from, which says that reddit posts are particularly smart and useful. Maybe just from their own marketing, which is repeated over and over in social media by these 'smart' guys?! It's completely not reflected by my practical experiences at all.

When I accidentally end on a reddit thread and read some posts, I'm always like "Well, and that's it? What a waste of time..."

> I don't know where the hype comes from, which says that reddit posts are particularly smart and useful

They're useful because there's comments from actual people.

I do site:reddit.com queries when I'm looking for product (and software) recommendations because otherwise you just find affiliate spammers (like cnet, wired, nyt and forbes).

Okay, for that particular purpose, I can well imagine that you are actually right. I usually don't even try that, and when I do, it's exactly as you say.

Maybe this was true in the past, but there are plenty of bots in reddit these days, and astroturfing/promotional content, etc.

While I still mainly use Google to search for terms online, I am increasingly using the free version of Perplexity for more advanced topics or general questions. Perplexity is an LLM like Claude and ChatGPT, but instead of relying on the data it's been trained on, Perplexity gathers a whole bunch of sources (websites, youtube video transcripts, etc.) related to your query, and then uses the contents of those sources to generate an answer. So while it may not be as smart as Claude or ChatGPT on certain topics, it does seem to hallucinate a whole lot less. And at times when I'm not given the answer I'm looking for, or when I want to make sure it's not making things up, I simply browse the list of sources it used to generate its answer.

https://kagi.com/

I've already mentioned Kagi.

I’m personally not liking Kagi anymore. There are a few bugs that I’ve been dealing with for a year now that haven’t been fixed.

I’m using Safari on iOS. After you make a search and you want to modify your search text by pressing the search bar, the cursor doesn’t show up at all. So when you type, it either doesn’t type in anything or it types in the middle of the search words, if that makes sense. You don’t know where the cursor is until you type.

Just those little things makes me not want to use it.

And the search results aren’t all that great either and I have to depend on Google sometimes.

It’s hit or miss but when I expect Github results for a repo, it doesn’t show up unless I add “gh” or “github” to make it show up. It’s not like I ranked github lower, it’s actually in the normal rank setting.

> It’s not like I ranked github lower, it’s actually in the normal rank setting.

Why not just raise it?

What iOS version are you on? I’m on latest and can see the blinking cursor when modifying a Kagi search query in Safari.

Reminds me my Nebula experience.

I watched a 120min video about LHC or something and missed a few words from a narrator. Pressed left to rewind, only to find out that it doesn't work. Their seek bar didn't help either by not showing some crucial information (don't remember which now).

As a result, I clicked nearby and got somewhere 5 minutes back with no chance to return or navigate precisely. "Hit the right pixel or get screwed" type of UI. Closed the video and canceled my trial.

They always screw up in little places like that. That itself is okay, everyone makes mistakes. What's not okay is that they don't even try to use their own service seriously. For a paid service, could at least license the player from pornhub (the guys who know their job and the importance of a video player on a video platform).

I’m afraid there’s no silver bullet.

Marginalia can be useful, even if the index is quite small compared to other search engines: https://search.marginalia.nu/

What are you trying to find? It's not the quality of search that has declined but the quality of the web itself. What used to be interesting blogs are now ad-filled SEO garbage. Good content is now more and more behind logins and paywalls. So much of the rest is bot generated.

Search as a concept is useless unless you know exactly what you are looking for and know that it will be in the top 3 results (for example a company's official website or a wikipedia article).

If you're getting a ton of irrelevant results, it's helpful to find a word common to them and exclude it from results; I don't have an easy way of measuring how many relevant results are excluded but if you choose it well it'll help. Similarly, find some list of all the operators for a given search engine and try them out.

I've found searching multiple indexes can be helpful, especially ones that seek different categories of thing out. For instance if I can't find something using Google, Marginalia can be surprisingly helpful. A list of sites to try is available here: https://seirdy.one/posts/2021/03/10/search-engines-with-own-....

Also, often finding some discussion of something that seems close to what you want and might discuss it briefly and seeing what it links to will help. But that's from mostly looking at academic stuff, which has copious citations that often won't show up in a search query for a term, but can be found in the likes of JSTOR or LibGen/SciHub.

Also from academic stuff: if you can find directories of things related to what you want (eg 'old books about the Standard Oil Company of New Jersey,' to take a real question I had), often a more general term ('old books') and then searching those websites (HathiTrust, Internet Archive, etc) directly will find things.

But that's for someone who primarily runs into trouble because I'm researching obscure subjects and probably are less applicable in other areas.

StartPage scrapes Google but manages to get rid of a lot of the noise.

Pretty sure they're paying Google for the data and not scraping it.

Oh man, I feel you. Searching for good info these days feels like panning for gold in a kiddie pool—every once in a while, you strike it rich, but mostly, it’s just wet disappointment.

I’ve been down the same rabbit hole (tried Kagi, got sticker shock, and yep, LLMs are great... until they start making stuff up). One thing that’s helped me cut through the noise is sharpening my critical thinking skills—it’s like giving your brain a search engine upgrade. I wrote a newsletter - just started - post about this (with some hopefully helpful tips), so if you’re up for a quick read, check it out: https://bottedconversations.substack.com/p/critical-thinking....

Let me know what you think, please! Always looking for better ways to avoid the search engine struggle bus!

Pay. Simple. The enshitification is a direct result of the free business model, where the users of the service are actually the product, and the advertisers are the real customers. Even if kagi feels too expensive, you are still voting with your wallet by choosing a service that you pay for and showing companies that they can focus on building good products for the people who use them and are willing to pay for them. FWIW, the same is true of social networks, but they suffer from the additional complexity of needing enough people to use a service to make it worthwhile.

search operators (https://ahrefs.com/blog/google-advanced-search-operators/)

in particular: site; intitle; - (to exlude search terms); * (wildcard matches); before/after:

In addition on Youtube using any search operator also seems to get rid of most of the "for you / you already watched.." crap that they for some reason decide to shove into the middle of the search results.

I regularly use Ecosia, Dogpile, Qwant. Saving D.D.G for simple tasks. Wikipedia displays a Timeline List with active engines highlighted https://en.wikipedia.org/wiki/Search_engine

Is there a specific reason why I'm seeing so little mention of DuckDuckGo?

It's not perfect, but it gets the job done 99% of the time, and I feel much more trust towards them than the alternatives.

I still need to figure out how to optimally perform the classic convenience search "X near me" without relying bigger names.

Could you provide example search queries that return bad results? My solution is to ask generic stuff from LLMs (and rarely youtube) and read specific on vendor's website. HN search works good as well.

Google still works for me. I don't know if it's because I am in the EU or because uBlock Origin gets rid of the promoted results, probably both.

Also, if you are searching for something time-specific then "after:" and "before:" work great.

I don't have a good answer, but we sure could use (a new) +Fravia in this day and age.

Try brave search.

Bing is now better than google.

You're right, though, it isn't quite as good as old google.

use chatgpt to search for you, it has this feature of searching the web and giving you relevant links so it won't hallucinate on links.

With links, it still seems to hallucinate on my end. Some info isn't even mentioned in the links it referred to, but it just spits it out like it's actually there.

interesting. For me it gave me accurate links that were tucked around third page on google search but were actually relevant links while google's 1st and 2nd page were filled with seo spam.

Oh no it still hallucinates - I searched for song lyrics I know exist on Google then bing, then gpt after trying to push it in the right direction and it insisted it was another song. I finally just started to manually look for it and found it.

Reply on news.ycombinator.com