There are a whole host of factors behind this, but I'm certain that the switch to Natural Language Processing / Semantic Search drove this decline.
“Product vs. Product” -> list of spam websites that just show them side by side with ads.
“Product Review” -> same as above, spam.
“Product referral code” -> sites with spam and no codes.
Interesting web content is in Reddit (and youtube), Google exists as a tool to search Reddit since the “new” Reddit site is awful.
Programming questions are mostly stack overflow, but occasionally you get a useful blog.
General knowledge is Wikipedia, news is axios.
Amazon for most purchases (except for products that have their own brand and sites that are at risk of being faked). Etsy for boutique stuff.
Most of the rest of the web is junk.
I think we could probably go back to curated 90s era web portals and have a better experience.
Agree, the web is unusable without an ad blocker.
If I ever X vs Y in google, I append site:reddit.com
I'm not a huge fan of Reddit, but it's about the largest forum. I really miss Forum search in Google.
You may find this useful: https://boardreader.com/
I try to "collect" search engines, so I'm always interested in finding more/sharing the ones I've found.
You can see the list here[0], and if you see more feel free to send them my way!
As others have mentioned, Google no longer respecting literal search terms has made it much worse for many types of searches. DDG had been great at this, but sadly has been following in Google's footsteps the past few years.
After reading through this discussion, I am inclined to explore search.
This crap sneaks up on a person one little adaptation at a time.
Searching on Yandex was a wake up call.
I update whenever I remember, which is sometimes.
It is currently up to date, except for the above linked engine. It doesn't have as many special purpose engines as I'd like though.
Added to my playbook. Thanks again.
I absolutely hated when they did remove it. I was able to sort of mitigate it by adding "forums" and "discussions" to search terms but it is not the same of course.
Sinister twist though: the SEO crowd has gotten wise to this. I've noticed recently that they've started throwing "reddit" into their spam as a random keyword.
As they wound down the top writer program, which is really where the "hits good" seeds are, they decided to pay people to ask questions.
Signal to noise ratio began to trend toward unfavorable. But traffic shot right up!
There have been other decisions contributing to the Quora we see today, but the paid questions really had the most impact in my view.
That said, yeah. There is still a lot of high value contributions to Quora. They are just a little harder to find now.
One thing I feel they missed the boat on was the credits system. Early on one could accumulate those and use them as a sort of currency. I "spent" some of my pile asking specific people for insight and it worked out well. I had others do the same with me.
Basically, one could get to a domain or subject expert and the site was still "family", so it would lead to a high value exchange.
For a little while, their text question UI hinted at what a Quora could be. Well connected Quorans could toss a question in via SMS and get solid responses back, sometimes quickly.
(Came down to follower counts, writer status, and a few other things)
These options were not abused much that I could tell and they hinted at a means to query people without draining said people, and or everyone participating having some give and take.
Was a fleeting moment, but one I ponder from time to time.
I would subscribe to a "group" that has the UX and overall dynamics of something like what I just described right.
I prefer "site:reddit.com" but honestly it's only because I'm used to it. I search in-app on mobile frequently (which I assume is the same as the on-site search) and it's generally pretty good. It usually finds something for me but sometimes using an external search engine works better.
Not trivial by any means but there's a lot of prior art floating around you could pull from.
Sure they could pull API access and be more hostile to scrapers but that's the current escape hatch for people who would have left because of the redesign.
Also love being redirected in order to continue reading certain threads
No broken links and much better experience.
What I've discovered, though, is that what google images now does, is it is only returning a few results. If you click a result, then there's a "new-to-me" "related images" section that shows images similar to THAT image, only in that window. It is in these images that I actually see the results I was expecting to show on the main page.
Its still far worse than google images 10 years ago, but not as bad as the UI makes you believe by looking at the initial results page.
Beware though, yandex allows NSFW by default. You have to enable safe/family search. This is the opposite of Google and Bing, so it could get you in trouble at work. Those crazy Russians.
Is it safe for cats to eat X?
Cats are mischievous and we love them.
X is not typical cat food. Let's go over some background on X before we answer the question.
Cats are obligate carnivores.
Thanks for reading, make sure to subscribe or buy these products!
The answer to the "can cats eat X", of course, can't be at the beginning or at the end. The reason is that the user must stay a long time and read the text, otherwise Google punishes the website for having a more than acceptable "bounce rate".
Putting the answer in the title (what has been dubbed "Anti-clickbait") also makes the site "less clickable" and will make it drop from the results. Trust me, I tried.
This is why we can't have nice things.
Old-school useful content is rarely monetized, just someone sharing their passion for something. Occasional affiliate link, with the obligatory apologetic "hey web servers cost money, so I included some affiliate links here!"
The uselessness is just a side effect of Google directing you toward paying customers.
The reason Google keeps those results near the top is because it drives up the cost of AdSense, since multiple competitors are fighting for the first page of organic results with those tactics, and AdSense is a way to "cut the line".
The result for users is that the first page of most result pages is littered with advertisement, either via real AdSense or with those inane marketing blog posts.
Easy for me to believe. They have to use some formula, and as soon as they change it, well, there's a massive industry dedicated to getting around it. If their algorithm is just "filter out what's useless", that's AGI.
Personally, I swapped to DuckDuckGo in 2019 and have been consistently happier but Bing or whatnot is probably equally valid at this point.
[0] When I search "postgres array_agg" on DDG - something I actually had to search for today - the Postgres documentation is the 6th hit (not even visible without scrolling down!), preceded by crap like https://archive.is/7JeSe. On Google it's the 2nd hit, also preceded by what seems like SEO spam.
In this case, DDG happens to have a bang shortcut for postgres which takes you directly to the search results on the PostgreSQL site if you query !postgres array_agg.
There's a surprisingly large number of these which end up making it more useful than the generic web search results page overall.
I actually switched to DDG cold-turkey and didn't use !g at all. Until I couldn't solve some problem and someone in my team said "it's the first result on google", I felt pretty stupid then. Since then I've given up and just automatically stick !g whenever I'm searching anything programming related.
Full text search isn't, but ranking the results is - and more importantly there is no objectively best ranking.
I doubt a million is going to make a dent in it. Content marketers are automatically generating this stuff, and Google has billions of users, many of which will occasionally invent a novel query.
try it here : ssh kiosk@gemini.circumlunar.space
https://www.aspca.org/pet-care/animal-poison-control/toxic-a...
In addition to plants, common human drugs like alcohol, THC and caffeine are also poisonous to cats.
Also common human spices such as salt, garlic and onion can be toxic too. Cats tolerate very small quantities of these (e.g. small slice of salami), but feeding your cat a chicken with garlic sauce is probably a bad idea.
Can cats eat chocolate? Yields ovrs.com (Oakland vet), purina, webmd, thesprucepets.com (cat blog run by vets), pdsa (pet veterinary charity)...
Can cats eat grapes? Yields top-N dangerous foods for cats listicles from various sites such as pet insurance and cat food brands. A response from a veterinary trust is in the top ten.
Can cats eat paint? Yields all reputable medical sources for the first several hits.
I’m not sure I can square this with your description of ubiquitous spam swamping out useful information.
Huh, I just tried this with a bunch of stuff and the snippets for each of the top several results for each all had fairly direct yes/no answers with reasons. Don't know if I got lucky hitting the right food items, or if it's a personalization issue.
can cats eat asparagus
I tried Google search, and produces a semi-useful snippet at the top, and reasonable results afterwards. Can you share what search you ran that had very bad results?
Can cats eat asparagus? ... It is neither toxic nor dangerous for our cats to consume in very small portions, but neither is it truly beneficial to them. Cats are obligate carnivores. Unlike dogs, who can and do eat everything they can wrap their jaws around, cats tend to be much more finicky eaters.
Can Cats Eat Asparagus? - Is It Safe For Cats? - ExcitedCatsexcitedcats.com › Blog Feb 12, 2021 — Vegetables, including peas, carrots, and asparagus, are safe for most cats to eat in small quantities. However, remember that your cat isn't going ... Interesting Facts About... · Which Vegetables Can I... · Is Asparagus Safe for Cats?
Can Cats Eat Asparagus - Cats Dogs Blogcatsdogsblog.com › can-cats-eat-asparagus Oct 18, 2020 — Although there are benefits for giving your cats asparagus, the potential risks are far much dangerous and outweigh the benefits. It is therefore not ...
Why Do Cats Like Asparagus? (And Is It Safe?)betterwithcats.net › why-do-cats-like-asparagus The short answer is that yes cats can eat asparagus in small amounts without any problems. But while asparagus is quite healthy for humans, your cat really doesn't need to eat it.
I'm honestly such a huge fan of the Bing widgets, every time I see a someone google search for something basic and need to dredge through all of the AdWords laden blogspam, while I know bing has a good widget in your face widget with the answer, I can't help but feel confused at all the "Bing bad Google good" rhetoric.
Some of my favorites:
Annual Weather charts: See chart with monthly breakdown of temperature & rainfall for any location along with record temps, days of rain, and configurable units; Google has something similar but it presents the first few rows of a table first, with a separate tab for charts, and then the charts don't have nearly as much data (avg hi/lo, inches of rain, hours of daylight (bing doesnt have hours of daylight, so +1 to google for that))
Random animal fact cards: Try Binging something like "marsupial", you get a beautiful hand-crafted info card. There isn't one for every animal, but it's clear thought goes into creating them and they always make me happy to see for whatever reason.
Contrary to that I tried "can cats eat salad" and got basically a yes in the first line:
"If your cat loves lettuce and greens, these can be a great source of water, bulk, and nutrients..."
It fits with the 2/3 of searches end without a click though. That was on Google's results page - I didn't have to click.
Usually: voted on/curated by members who are specialized in some way (reddit, so, hn, wikipedia to a degree).
The fallbacks are editorial sites.
If you make a search engine that focused on indexing user-curated sites + results outside of that that are themselves curated/voted-on by your search engine users (ie, you and others could upvote axios and Amazon as a good source), I think you'd have an interesting model. Basically, take the HN/reddit model to search itself.
Which is easier: enforcing one account per person and then identifying bad-voters, or not doing that and having N pages, and N*M page links?
Further, you'd now have direct access to customizing results per-user. Use my votes.
You are absolutely correct. So many of the top results are basically machine-generated pages, perfectly optimized for search robots, not so much for humans.
I tried this with a bunch of products ripe for spamming: instapot review, nvidia 3070 review, and Apple Watch review, and the first page results were almost entirely reputable.
I also tried “huggies vs pampers pull ups” and the result is quite a good variety of forum threads, blogs, and reputable articles. “Quasar formation” leads to Wikipedia, but also academic articles and astronomy.com. “How beer is made” is even better quality.
Is most of the web junk? Have I gotten astronomically lucky in not finding it? Are these somehow the exceptions that prove the rule?
(Ironically I think my search terms caused this comment to be flagged as spam..!)
It's less common stuff that tends to give you mostly crap.
Google today limits how much of "the web" (their index) they will let you see.
You are only seeing what they choose to allow you to see.
Google, not the user, determines "relevance" and Google automatically excludes results. in theory this sounds useful. In practice, Google is now limiting max number of results to 200-300 or 500 if you add &filter=0. Retrieving 501 search results for a single search is not allowed. Sorry.
Try a search for some common phrase like "the web". Surely this phrase occurs on more than 500 web pages. Yet Google will limit you to only 231 results. Does that represent the entire www. Then you try "repeat the search with the omitted results included". Google then limits you to 466 results. WTF. What if you searched page titles for some string. Is every string you search going to be found in less than 500 pages.
Google search results today are not representative of the entire www. Not even close.
I’ve found this to no longer be the case after Reddit has starting doing nefarious tricks with the date of a post.
For example, I’ll click on a link in Google showing it’s from last week when in reality the post is 2 years old.
Probably some financial positive feedback loop is responsible for this.
That's true. But out amongst that is the result you actually wanted that google doesn't have any incentive to lead you to.
I think the future of search is in finding trusted results, such as those upvoted by reddit or HN or SO, assuming the upvoting system is robust enough to withstand spambots etc. The question is whether a big enough fraction of the population actually care to get good search results.
I didn't realize that I'd migrated over to this. All my searches now are 'xxx reddit' or 'xxx stackoverflow'
It's only a matter of time.
1) Bring in tabbed search result like Cuil search used to have. Easily lets you go to similar/variant topics from UX POV and must be a good way for a search engine to learn what people want more.
2) Add a 'dont show this site' option. So they can easily see when people get annoyed with a result and use that as a ranking signal against search terms. E.g. if people press that for a particular KW but not usually Google knows they got that result wrong and if people press it for almost everything you know its a site people dont want to see. And at a personal level its great to custom remove stuff, like when Pinterest was dominating so much a year or so ago.
Would be a great way to community curate results.
Once I took the time to read one of the sites and it’s obvious it’s generated content, and awful at that. I still reading pros and cons of one of the product and the same thing was mentioned under both, just with different wording.
GPT-3 or similar is going to make this way worse.
The only time I still use google (via !g on ddg, what an awesome feature) is to search for local info or in native language. But even here ddg is getting better, especially if I append the name of my city or country.
Back in the 90s, search engines were driven largely by sparse vector representations of the documents such as TF IDF vectors before latent semantic indexing, topic modeling, and other dense vector representations like sentence embeddings entered the fray (not to mention non-content features that use the web graph, click stream, etc.). A lot of NLP applications use a mix of dense and sparse features but it's hard to get the balance right in a way that works for all inputs. Google's pendulum has swung too far in the "dense" direction, as it certainly seems "dense" a lot more often lately!
I think it's a popular opinion on HN that Google search sucks, but I just don't agree. I used DDG on all my devices for the better part of last year, and bailed when I noticed by g! usage ratcheting up.
While it’s an unpopular opinion on HN, there’s no denying that from a user perspective, that’s only a good thing.
When people are trying variations on a search it's a clear sign of failure. They should do something about it, like use a verbose natural language interface or select a different strategy for ranking or enable the exact in-depth mode. Apparently the NLP community can do natural language Q&A in papers, but Google can't do it in search.
Other pain points: searching in depth all results on a query, not just the top skim and remembering context between searches.
And Google Assistant itself is too poorly integrated and dumb. Where's the GPT-3 like language skill? So many TPUs what are they doing all day long? They have more text and images than any other entity on the internet, Google bot has been sucking it up for so long (not to mention links, keywords and clicks). It should show in the quality of their AI. It's a shame to have OpenAI steal their thunder like this.
Undeniably they are the largest, most successful at search and they spend huge amounts of money on it.
And google is probably the biggest producer of ML research including? including a TON of NLP as it's a driver of their profit.
The web of 2021 sucks.
At least there should be a checkbox for "make a best guess when limited results" / "give me my exact search query" (maybe there is somewhere & I've missed this)
There is, one of the drop-downs gives you the option of "all results" or "verbatim".
It doesn't actually work, of course.
Which is ironic, because one of Google's original innovations was to make a space an AND connector instead of OR (which its competitors used to maximize result counts). Back then, they understood that fewer, more specific results were better.
"when the big Internet search engines like Altavista first came out, they bragged about how they found zillions of results. An Altavista search for Joel on Software yields 1,033,555 pages. This is, of course, useless. The known Internet contains maybe a billion pages. By reducing the search from one billion to one million pages, Altavista has done absolutely nothing for me.
The real problem in searching is how to sort the results. In defense of the computer scientists, this is something nobody even noticed until they starting indexing gigantic corpora the size of the Internet.
But somebody noticed. Larry Page and Sergey Brin over at Google realized that ranking the pages in the right order was more important than grabbing every possible page. Their PageRank algorithm is a great way to sort the zillions of results so that the one you want is probably in the top ten. Indeed, search for Joel on Software on Google and you’ll see that it comes up first. On Altavista, it’s not even on the first five pages, after which I gave up looking for it."
- https://www.joelonsoftware.com/2000/08/22/three-wrong-ideas-...
A lot of people say this is because Google is worse at returning the desired search results for various reasons (algos, ads, spam, etc). I've come to suspect it is more deliberate on the part of Google.
I've said it before, so I'll belt out the chant again: "Google doesn't make money from providing the right search results. Google makes money from keeping you searching for the right search results."
This is especially true for the vast majority of people on the internet who do not know there are any (better or worse) alternatives to Google.
That felt like a thick wall of glass separating worlds have suddenly come into my view.
I guess it's time for something new to do the same thing to Google that it did to those companies years ago.
I'd much prefer going back to something like that now than bother with Google's approach; depending on the search term, I already know what the top sites are going to be, and I know they won't have what I'm looking for.
This feels very true... the majority of my searches go like this:
1. Search with all relevant terms with some parts that should be exact combinations in quotes => nothing or BS results
2. Remove words that Google is fixating on without context => no or seemingly unrelated results without search terms at all
3. Reduce specificity to 2 or 3 words, topic and subtopic (trying to locate context only and search the rest myself) => sometimes ends me on sites where I can then just browse for the result i'm after
4. Worst case reduce to single search term to find huge context sites and manually search myself. Sometimes at this point I just give in and manually navigate to other sites where I know I can manually browse and narrow down context myself and then try to find what I'm looking for... it really feels like a curl web scrape and grep would work better than Google at this point (yes I know google "site:", it's doesn't work properly anymore).
I can't say I've really counted how many days that's true. I can't even say that I really remember what searches I did yesterday or the day before. But if I'm not totally weird and google searches were dramatically better in the past, then they must have produced the desired results every time on almost every day.
Same for me, but that's because I just stopped searching for things that I know will not give me good quality results as they used to be circa pre-2010 back when search tools like +, -, and "" still worked and the results weren't filled with generated SEO texts.
Oh, I forget some people doesn't know why Google enjoy their current position:
Back before 2007 they completely blew competition out of the water:
If something was accessible on the Internet and wasn't behind a noindex spell, Google would find it.
Compared to other search engines that both then and now work more like Google does today it was totally amazing.
Then things started to go sideways:
- first there was: did you mean <something else with similar spelling>? (this was actually user friendly)
- then there was: we didn't find many results for <search term> so we included <related but different search term>, use double quotes to search for "<search term>"
- then there was fuzzing: expnding all my search terms into the unrecognizable unless I double quoted them
- and the latest few years they have also ignored my double quotes
somewhere in between there they messed up the + operator that used to mean "make sure this term is included" as well as ~ that used to mean I wanted Google to fuzz that term.
Sometimes I can get better results by trying to think how my wife would phrase the question, i.e. instead of searching for
- <search terms including a weird mispeling from a dialog box>
I search for
- <why does my computer show search terms including a weird mispeling from a dialog box>
But other times nothing works.
IIRC this one was driven by Google Plus marketing wanting "+" to mean "Go to this page on Google Plus" so they could do some co-marketing thing with +Pepsi.
It never made sense to me for Google+ but it broke that feature.
Lately it seems + might be working on DDG and maybe Google as well, only I have to use it in combination with double quotes like this:
+ <some search terms including a +"weiiird" one>
Edit: Can't say for sure if it is a feature or if I just knock out the dumbifier somehow :-P
https://saraacarter.com/google-whistleblower-state-attorney-...
This should destroy Google, but for some reason it doesn’t, and it baffles me that they are still surviving in this.
I suspect it is because the best sources of information do not serve adds, but news, spam, and social media do.
I don't blame Google for this. 99% of alternative medicine is quackery and it's almost impossible to separate wheat from chaff in an automated way.
I only want Google Search to filter based on search terms, not it's creators opinions about what I'm searching for.
If I'm looking for a review of a Yeah Yeah Yeahs album, I will see links to Amazon, Pitchfork, Rateyourmusic and other sites well before I find something written by a regular person on their personal blog.
Same goes for products. I will see a page of Amazon links before anything on say, a boutique Ecommerce website with a Shopify backend.
https://www.google.com/search?q=See+them+more+than+my+friend...
I don't know what other people see, but I just get SEO garbage listicles. Nothing even remotely relevant. Add "lyrics"? You get lyrics..... for other songs! https://www.google.com/search?q=See+them+more+than+my+friend...
You have to quote the entire string! Which won't work if you misremembered a word or two.
For those who don't know, after a search, click on "Tools", then "All results", and select "Verbatim".
One word of caution: I use verbatim mode and I regularly (but not always) get Wikipedia clones as my top result. It must be skipping some of their anti-spam filtering.
I don't understand what goes on corporations. I guess power users aren't a target demographic anymore.
* shows results despite having zero actual matching results
* ignores parts of the query even if they are surrounded in quotes
* gets completely thrown off by keywords (and, or, etc) and symbols (=, ;, etc)
But this needs to be public effort to create any meaningful action from search engines.
How many of you would be willing to contribute to such platform by submitting your bad search results?
[1] https://needgap.com/problems/207-search-engine-wall-of-shame...
I thought the main issue here was Google directing searchers to its own services and scraping data from non-Google websites to present on its own pages? It's declining search result quality is certainly an issue, but I don't think this link is about that at all.
I don't think it has a lot of to do with new search algo. I think it is the opposite, the algo is mostly the same as a decade a ago, but now it is abused by SEO expert. And Google doesn't seem to care yet
Specifically, it's handy to have the query, and the URL that you think would be the best result.
Google engineers can then peer in and see why that URL didn't rank highly.
The most common reason is it's behind an invisible captcha or blocked by robots.txt, so do check that.
Lowes has made a video, so have dozens of contractors. It's amazing how tool rental firms have not advertised on there ... actually that is a bit weird...
SideNote - for some reason I replied to an email outreach from Rand, and he actually replied back within a few hours, clearly having read my words and come up with a real answer.
I was frankly shocked. Either he works an inhuman amount or he has secretly invented AGI to respond to his emails for him.
Kudos and good luck
Oh, and then there's a bunch of weird links that come up to obvious content generated by AI being peddled. Odd.
[0] https://en.wikipedia.org/wiki/Search_engine_optimization
I have ZERO doubt in my mind that a Google search update that decreases their ad revenue would be rolled back and metrics tweaked to make it so. So...
Mostly ignoring what is on the sites
It's the switch from lexical/syntactic search to semantic search. There are benefits and drawbacks to each and it would be nice for Google to give you the option to try each.
I wager that most people don't perceive a decline in search results quality. At least not to an extent where they'd notice and switch to using Bing.
Google giving up on difficult searches and focusing on showing promoted links, AMP sites, and "rich" result boxes is probably a sound business strategy too, and is unlikely to make users switch search engines this late in the game (Bing/DDG don't seem to do better anyway).
So GP may not be "an unprofitable long tail". GP may in fact be "a profitable typical long tail for whom Google Search is frustratingly unhelpful".