The Tragedy of Google Search
theatlantic.com
theatlantic.com
- Google Search has evolved significantly from its early days as a simple list of blue links to becoming an encyclopedia, predictive engine, image repository, shopping mall and more that is overloaded with information.
- It has become more difficult to find authoritative answers on Google Search due to an overabundance of sponsored content, prompts, and low-quality keyword-stuffed pages.
- Google is currently undergoing an antitrust trial regarding whether it maintains its search engine monopoly through anticompetitive means such as exclusivity deals rather than having truly superior technology.
- Access to large amounts of user data is very important for powering search engine algorithms through personalization and improving the user experience.
- While Google argues diminishing returns to scale for user data, internal emails show Google engineers acknowledging that scale remains highly important for the company.
- Some feel Google has lost its way and become conservative due to its success, with its corporate bureaucracy stifling acquired companies.
- Google Search has become bloated with advertisements and prioritization of its own services over organic results.
- Search quality may be declining as its algorithms are gamed by low-quality sites and search engine optimization techniques.
- Google's trajectory of scaling up its mission to organize the world's information through acquisitions and exclusivity deals has put pressure on it to keep growing.
- No company can likely grow indefinitely to keep up with the ever-expanding internet without being overwhelmed, as Google Search now demonstrates.
No, it doesn't. It has one master: advertisers. Advertisers are the customer. Users are the product. Search results are the bait.
This is self-evidently true. Your customer is the one who pays for your services. Google's users don't pay, never have, never will, because search is not a product you can sell.
User attention, on the other hand, is a very valuable commodity for which people will pay handsomely.
> No, it doesn't. It has one master: advertisers. Advertisers are the customer. Users are the product. Search results are the bait.
I wouldn't put it that strictly. Users are still masters in that they can continue to use Google or not.
The currencies are different but Google absolutely needs to satisfice two disparate customers.
Users are Google customers in the same way that (say) fish are a fisherman's customer. Fisherman engage in barter transaction with fish, offering them bait (food, or at least the prospect of food) in exchange for allowing themselves to be sold as a product to a third party who gives the fisherman actual money. That is what makes the third party the fisherman's customer and not the fish. Likewise, Google engages in barter transactions with users, offering them bait (useful information, or at least the prospect of useful information) in exchange for allowing their attention to be sold to a third party for actual money. The money is what makes the advertiser Google's actual customer and not the user.
When I was using Bing AI or Google Bard to summarize an article, they often refuse to do so because of paywall or some other reasons.
I'm not really sure how they lose that case -- Firefox switched to Yahoo after MS outbid them, and switched back to Google the second they could legally do so and keep the money.
When I think about what I actually use search for... it's highly situational. It depends as much on day-to-day and moment-to-moment context as it does any long-term trend specific to myself.
Whatever "personalization" could be realized over search results, would it really offer that much benefit over improving the search function to sift through and find results that are just more useful generally?
What I mean is there seems to far, far more to be gained from making search results better fit the query than to fit me personally.
I've had "Search Generative Experience" (AI/LLM synthesized result/summary above the standard web results) turned on for awhile. It is definitely the future. To survive the flood of revenue optimized content (recipes being a familiar example), you need an active [software] agent working on your behalf to read content and make one straight thing out of the crooked timber of the web.
Google started slipping when it began assuming that it knows better than you do. For instance, when the exact search operator (" ") stopped working.
I'd take the Google of 2014 over the Google of 2023 without thinking twice, and I think that I'm not alone in this.
[verbatim mode] https://www.google.com/search?q=unexpected+newline+in+compos...
[quotes] https://www.google.com/search?q=%22unexpected+newline+in+com...
The top two results are the same, but besides that verbatim mode seems way better to me.
(said the independent search engine creator with the same air of righteousness as when Plato argued philosophers should be kings)
Same situation with python honestly. No I don’t want to use pandas, if I wanted better data wrangling I’d just wrangle in base R then import the wrangled data into python if need be instead of adding another dependency to my python code. Apparently this makes me a heretic based on google search results.
Since I know the top results are mostly useless spams on some searches, I used to go to page 4 or 5 to find the actual results. Now to achieve the same result I have to click and scroll 5 times.
It still works.
Works exactly how quotes used to.
Actually, it has happened with results that include hacker news discussions. I scrape through and expand EVERY comment on a 6 month old post and still find no partial or exact matches for my search keyword. It is very clear to me that Google has switched behavior. It used to flex on how awesome it was to bring you to an esoteric corner of the internet matching your exact query. Now, it tries to converge your queries into what it thinks you want(not what you asked for) and funnel you into large websites at the expense of utility.
Which can be incredibly frustrating if you know the problem well enough to need a different answer.
I also love when my search is narrow enough that Google instead starts to return more generic pages for totally different programming languages instead of the smaller result set that matches exactly. It is very easy to observe as you explore a topic. At some point, you got a wall where any increase in search specificity seems to yield worse results than queries that helped you get to that point.
Which... Why isn't Google learning that some users are constantly seeking out verbatim for results they'll click on and adjusting their search results to be more verbatim?
Its results are superior, with way less spam and SEO optimized content.
a) there is no giant industry optimizing for ranking on Kagi. There are whole armies of people trying to game google search, meaning whatever heuristics they have would probably be rendered useless shortly if google were to adopt them.
And
b) Kagi has a much smaller, more technical userbase. So like the early web they can get away with a search experience tailored for those kinds of people and not millions of people powering on a smartphone as their first computer.
Kagi works for you, with your input. It doesn’t do “advanced guessing” by implicitly and opaquely processing your interaction with it.
I guess their "Universal Summarizer" is a part of their indexer, so they try to understand what you want and provide you more relevant results.
They probably have other signals and systems to detect spam and other unwanted things.
Combine it with personal website ranking and lenses, it surpasses Google easily.
Also since it has no ads, there’s no noise.
I don’t use Microsoft products, so I can’t comment on Bing.
I suspect this is a niche feature for power-users only. HN is filled with power users, so they love Kagi, but it doesn't necessarily translate to the masses.
I think it’s important to measure and quantify these things.
I can't provide you with side-by-side examples because I don't compile lists of such examples. I'm searching to find stuff, not researching search quality. I just know that I have more problems getting good search results from Google over a couple of the other engines.
If a user of service thinks a different service is better, then it is better for them by definition. "Better" is a subjective thing. If X seems better for me than Y, then it is.
"Are US drivers licenses forgery-proof?" and "Can US Real-ID drivers licenses be forged?"
gave utterly irrelevant #1 or #2 hits: nothing to do with US or Real-ID (zero mentions), it's about New South Wales, Australia (lots of mentions of "NSW"). (although from US magazine Wired.com, but Google can still tell this article is totally irrelevant).
2) I previously in 2022 posted "Google search relevance fail: result for “Africa longitude” [https://news.ycombinator.com/item?id=30337563] where the #1 site hit and text result were the grossly misleading "Africa/Coordinates 8.7832° S, 34.5085° E", which is not a range, or even a centroid, just a junk location in southern Tanzania, where a (Russian?) SEO spam person seemed to have dropped a bogus pin, mislabeled "оф. 3, 27 ул. Хабаровская, Bila Tserkva, Russia". In Tanzania. 17 months later and same gloriously unfixed garbage results.
Other bizarre top hits were the small Ottawa e-commerce digital-map site https://www.mapsofworld.com/lat_long/africa.html, with the incomplete and inaccurate non-sentence "Africa's latitude and longitude lies between 9.1021° N, 18.2812° E." (That e-ecommerce store seems to have an autogenerated page with incorrect information on latitude and missing longitude for each country).
These are cases where Google itself 5 years ago would have done better. Or compare to Kagi.
Some people admittedly expect pretty weird websites to exist. Like who would even publish a good objective comparison between products in a specific category. But people make those queries looking for those types of results, and spammers generate such content to make bucks off the traffic. 20 years ago you'd have gotten nothing in most cases.
Consider coffee. You'll find hundreds of articles comparing the most high-end grinders. You even have PhDs writing white papers on the physics of coffee and how particle size distribution achieved by different grinders affects extraction quality. Those are genuine opinions from aficionados.
The same is true for all kinds of other things. The internet has a lot more than the auto-generated marketing crap you give it credit for.
Sniper in mahwah for example will discuss some of this for example from an HFT angle.
So what do you think, will be the goal of all advertisers?
To get their recommendations baked into that AI. Direct bribing/revenue sharing with the manufactur of that AI. Or sneaking data into training sets via bribed employes and so on.
I don't trust black boxes and never will.
Paid search is the future. Either you pay a subscription fee to something like Kagi, or you pay for hardware and electricity turn run your own "AI agent" locally.
It's a negative-sum arms race, but if Google (or whoever) wasn't spending the money to fuel the arms race by passing on ad money, who would pay for Google to exist in the first place?
That's just my hypothesis, of course. I don't know for certain. But it does make me very skeptical about using AI to help search the web.
I want a robust set of search modifiers instead.
How long do you think it will be before that software agent's output will be just as riddled with commercialized content? I think we might have only a short window where we get to enjoy the fairly unadulterated output of the base LLMs by the large providers (yes, I'm aware that they're screening for safety already, but there could be a lot more coming). Imagine a dialogue like this: Q: What should I watch out for when buying new car tires?
A current answer might be something like: Here's a list of things to watch out for when buying car tires: A, B, C, D, E. Remember to always consult with specialists. Shall I explain more about the topic?
A future answer might look something like: According to experts like the professionals at [tire shop near you], here are the things you need to watch out for when buying car tires: A, B, C. You can use this coupon to get a 10% discount off your next purchase at [tire shop near you]. Shall I book an appointment for a free consultation at [tire shop near you] for you?
I'd love to have one, but do any exist? SGE works on google's behalf, collecting data for google who will use what they learn from it against me. I'm guessing that until we can use common hardware to train/host an AI agent ourselves, we're stuck with terrible search results unless we're willing to hand over more and more data to others who want to exploit us, while pretending we can trust them to return results that are in our best interest instead of theirs. That's not a great future to look forward to.
But this feels like too obvious a conspiracy theory and it’s probably not directly designed to act that way - more likely it is an unintended consequence of something else. Unintended, but tolerated all the way to the bank.
It kind of depends what the ad value per user is, and then charge X% above that. Though maybe it’s so high that it would be an enterprise price level.
But also, if someone claimed to be a Google engineer and told me they purposely rank up pages that have Adsense on them, I wouldn't call them a liar immediately.
If Google could drive higher levels of engagement and satisfaction by screening out the dross I think they would, it’s clearly within their abilities.
Hopefully AI gets good enough and scales efficiently enough so they can disaggregate this kind of ranking decision.
By putting fluff and ads you extend the time on the page making your site seem more valuable.
It is hard to tell "that didn't answer my question" from "that quickly answered my question wrong" after all.
(Note it isn't uncommon to always bounce off a recipe site as you check multiple to make sure the one you found makes sense)
How in the world can telemetry tell you any of that? The only metrics I've heard from Google that relate to it is whether or not they return to do the same search again within a short window, but I don't think that tells you users prefer SEO junk or that they're satisfied with the search results.
Even not Google. I wonder if they can still turn things around, but I have the feeling MS caught up finally at high speed and passed them.
PS I was reading (sceptically) 'google search is broken' already for a year or so in HN comments and have to admit I believe it now.
Even ignoring results, mainstream started to notice a while ago. Perhaps the biggest social proof at large is when it is shown distilled as comedy and people can laugh about it:
https://news.ycombinator.com/item?id=37606618
Also, searching for that quote on HN via Algolia gives many hits.
The #1 or #2 hit is utterly irrelevant: nothing to do with US or Real-ID (zero mentions), it's about New South Wales, Australia (lots of mentions of "NSW"):
> ‘Tough to Forge’ Digital Driver’s Licenses Are—Yep—Easy to Forge. Researchers found a litany of security flaws that allow simple, quick, and cheap forgeries in Australia. https://www.wired.com/story/digital-drivers-license-forgery-...
https://cis.org/Report/Americas-Identity-Crisis-Document-Fra...
But that link is a 2002 article at least about the US, but not Real ID.
(any random hit for ""real ID" forgery" in the last 18(!) years would be better)
I will also add that the particular link you showed is a 7300 word document from 2002, most of which has nothing to do with ID cards at all. It’s at a college graduate reading level and would take 26 minutes for someone to read according to wordcounter.net.
even if its only 10 searches a day, it needs something. I am never going to use it unless I can try it out for a while.
> “It’s absolutely not true that scale is not important,” Manber wrote. “We make very good use of everything we get.”
I think that is true, but then the obvious conclusion - if one party doesn't have 90% of the market then the quality of search results will be lower. Successful anti-trust prosecutions against the party with 90% of the market will have a likely outcome of:
1) Reducing the quality of service for everyone.
2) Pushing the 90% market company to be one domiciled somewhere with more reasonable monopoly laws.
Successfully proving that more data -> more scale -> better results is actually implies that one company with 90% market share is desirable. Although I'm still not going to use Google search, I don't think that argument is correct. But the line of attack is not clever.
With a natural monopoly, if you built 2 of them in defiance of the economics of the situation they'd have similar quality of service to the single operator.
> A natural monopoly is a monopoly in an industry in which high infrastructural costs and other barriers to entry relative to the size of the market give the largest supplier in an industry, often the first supplier in a market, an overwhelming advantage over potential competitors. Specifically, an industry is a natural monopoly if the total cost of one firm, producing the total output, is lower than the total cost of two or more firms producing the entire production.
In theory pricing/output in economics is always “quality adjusted” (which is one of the hard things in inflation series)
Yes, traditionally a natural monopoly is one where if you split them, costs go up. With Google, GP is saying if you split them up quality goes down. Either way there’s “consumer harm”. (I guess very directly if the time to search goes up, the economic price of using Google goes up too)
Disclaimer: developer
I am bootstrapped, don't advertise, and need no investment
Technical searches on Google have become a pain in my ass. I'm constantly hit with low effort articles about topics that Google has guessed are similar enough but definitely aren't.
10-15 years ago, Google didn't have the same internet to crawl -- but it also didn't fucking _ignore_ what I asked for.
I was there. We used Altavista and Yahoo and they rocked. Until we tried Google.
You are also misconstruing what I said. I don't say I "don't really know" if it was better. I don't think it was better and seeing people baldly assert the opposite without proof hasn't convinced me my memory is wrong. I'd like to hear something better than "trust me bro." I'm tired of people getting published with evidence-free arguments (though I guess it beats someone condescending to me based on a wrong guess of how old I am).
If you are looking for big search engines, there aren't many really, I strongly believe you should give them all a shot and just alternate between them.
The biggest takeaway from this is that google isn't massively better than the worst alternative anymore and it sticks mostly due to recognition.
At first it wasn't even because their search was that much better than anyone else's. What made google popular was that it wasn't a massive, messy, ad-filled page of things you didn't care about. Other search engines were bloated "portals" that tried to do everything: email, news, shopping mall, weather, etc
Google had a simple clean interface that loaded instantly on a typical dial up connection and that's really what got people to make it their homepage. Today, google has slowly turned into exactly what it was created to solve. Ad-filled bloated pages that try to be everything by pushing things you don't care about in your face.
https://search.brave.com/goggles/discover?goggles_id=https%3...
What do?
Google remains the best search engine, period.
Nothing is even close.
The one thing I have noticed recently though is chatgtp is way better for certain queries. Like I wanted to figure which brand of fillings I'd had in my teeth the other month. Googling "brands of dental fillings" was hopeless - it gives all results like "here are a number of different fillings, including: amalgam, composite fillings". No brands. Chatgtp though, first try listed all the brands. I think it was sonicfill by the way - quite cool tooth tech.
Just turned off its servers and said "we're closing".
Then some would realise what a huge impact it serves on a daily basis.
Pop culture will never last.
We're building a new search engine. Of course it's AI driven, but at its core, it's all about human dignity. When you use it, you'll have a voice. If you'd like to try it, please email me: josh@sirch.org
Thanks! Josh
PS. If you'd like to join our team, we have 16 people now (mostly Snapchat/Instagram/Twitter folks), and we'd like to go up to 50 before we raise capital. It's going to be a wild ride.