Which as some running a website raises a fascinating question. If Google is just going to crawl my sites and present information as an AI summary on their site, then what exactly do I gain by allowing Googlebot to crawl my sites?
This was an interesting dilemma because it was very clear that the money was way less than the loss in ad revenue due to traffic drop, but it was also clear that if we wouldn’t take the deal, a more desperate competitor would, which would result in the same traffic loss but without the extra google money. So the company took the deal.
History repeats itself here, with the difference that instead of paying for the data, the ai crawlers simply take it for free.
The search space is actually not that unhealthy. There are competitors. They just lack the reach of Google. These competitor will become increasingly useful when our personal AIs plug into them to search for the right info.
That said, I think the old internet business model is dying. Why will people write articles and pay for hosting if they can't receive any advertising revenue? I think I'm okay with this. Before advertising decimated the free internet, people ran sites for fun. Maybe we return to that.
1) because that fancy summarised results page will effectively be the ad. The results will push you towards google's partner sites or promote behavior benificial to them.
2) because partners may soon be able to pay to change the results of searches. Who won an election 20 years ago? That "fact" from the AI could depend on which answer has been sponsored by which political interest. What happened in china in 1989 ... how much money do you have?
3) the user becomes the product. No ads in search results, but our searches and online behavior tracked by them will be sold to whoever is most interested in what we do online.
>Google parent Alphabet profit jumps 81% in Big Tech earnings roundup
https://www.euronews.com/business/2026/04/30/google-parent-a...
TL;DR: Google is heavily stuffing more and more ads into every monetizable SERP, which explains the ever-increasing revenue.
But my theory (expanded in more detail here https://news.ycombinator.com/item?id=47957708) is that its cash cow is heavily optimized towards its current anti-competitive, multi-sided chokehold of the online ad market -- look up the AdTech Antitrust findings around manipulating ad auctions, e.g. Project Bernanke -- which relies on ad volume.
With agents however, I suspect that volume will shrink drastically even if they stuff ads at each step of the conversation. Because it's only the final click that will matter, and those ads will be meaningless.
It's not clear that Google can charge enough for that final click to compensate for the loss of both, the value of all the ads that lead to it, as well as their ad auction manipulation premium.
All the "AI search" I've tried is far less relevant than a proper search, using search engines without it seems much faster and more relevant.
Whenever I get AI search results, its summaries lack substance, are often just plain wrong, and so on. When I check, it's often usong the wrong aggregate data to return results.
Search engines these days, traditional ones, have been hobbled by targeting the lowest common denominator. They drop search terms, alias them (eg, Joe=Joseph,Josephene, Jodiene), and thus return endless bad results.
Using "web" search on Google, for example, putting all search terms in quotes, helps enormously. If searching for Joe Baker, you don't want aliased responses.
My point is, AI is yet another abstracted layer. It not only aliases things, it halucinates, and further, it doesn't even show the context it simply summarizes.
It's the next layer of crappiness.
As a side note, there os a product in my home country called the Swiffer. It is like a broom, but uses disposable sticky paper instead of broom bristles and a scoop.
Point is, it isn't very environmentally friendly. I weirdly see environmentalists using it, instead of just a good old broom.
And now I see environmentalists using AI for search. What? Why! The sheer additional power and cost, all the new datacenters, hardware, just for what??
Even of one believes it is mildly better (which it isn't, if you use traditional search properly), are you now replacing your broom with a swiffer?!
The amount of environmentalism that goes out the window, for convenience, astonishes me.
The real test: dont test AI by asking it questions you already know. Ask it a proper question and then go out and find the answer yourself. That is where AI really falls down, the non-trivial stuff.
Several months ago, I was getting hallucinations and silly answers, like Jaco Pastoreous being the bass player for Metallica.
but lately it’s been great. Especially for technical syntax stuff, like asking for syntax on a jq command, or sal syntax, or something trivial like movie casts members….
Edit: I also assume there’s now some pre-processing filter to retrieve answers from a cache or something, because I’m getting pretty long answers very quickly….
For average results, that can have an acceptable answer that tends to the mean or societal mainstream views along some subject, the google answer works for me. For others, which are well unfortunately most of my queries these days, the google answer is brainwashing and propaganda. It feels little more than a continuation of the SARS-COV-2 era fact checkers which were actually failed attempts at propaganda machines for enforcing certain views as true while others were cited as "evidence does not suggest" (hand selecting the evidence of course means this is nothing more than confirmation bias).
I think the AI technology used in most LLM models is an acceleration of this- it will tout mainstream views or hand-picked views as truth while calling others false. And without an ability to search any more for opposing views, you get left with a really warped view of reality that might work for most. Though not for everyone.
I've had the opposite experience with Perplexity Pro. Yesterday it took just under nine minutes to deliver a detailed and accurate answer as to the nearest locations where I can buy Meta Ray-Ban Display glasses (at least 3 hours drive away FWIW) that day.
In the past several months I've spent countless hours using search engines with dismal results that weren't close to that.
If conserving energy was important, none of these AI datacenters would be built, none of our items would be shipped across the planet from Asia. They would conserve energy and import raw materials only.
But! Does Musk have his robot army or not at this time? That will determine how the pitchfork mob turns out.
In this context, if Google is going to give me the recipe without having to scroll through the story, that seems like a win to me.
The ad-revenue driven Internet of web 2.0 is finally dead and I'm not sure I'm all that sad.
But then, the sites it's training on are starting to do the same thing, so maybe it won’t matter. Just last night, I pulled up four sites with “gluten free almond cake” recipes. One specified less than 1/4 the flour it would have needed, and another didn’t have any butter in the ingredients list. I had to eyeball the median and tweak from experience to actually get a bakable cake out.
Oh, maybe I should drive to the major city and check several bigger bookstores?
Or order it online? But that would be from the third country because this one is weirdly blind to variety. That will be €20 for a book and some €10 postage.
You're sure I can just get my recipe from the store?
Later and you start seeing "Use this new processed wunderfood ingredient substitute!" (70s-00s)
Local bookshops are great, use them or lose them.
The only solution is to find recipe books that were printed in previous decades.
Which is ironic, given that Google's entire value proposition (to users) was extracting signal from noise...
... and now it's come full A/B-advertising-optimization to being useless at that, when the need is greater than ever.
Imho, Google's greatest failing was missing how its own incentives warped creation of new web content, and failing to account for that strategically -- it turned the web into something it can't itself usefully parse.
And almond flour does its thing by carmelizing in combination with butter and sugar, turning your whole cake into a sort of giant macaron. You can’t pull it off without any one of those things.
No. It may be good for the consumer right now, but not ultimately (and on top of that, I would argue that reducing everyone to a consumer is already the wrong framing -- you need to ask what's good for the citizens). Having ten competing supermarkets with various interweaving supply lines is ultimately much better than having one giant supermarket, because that one monopolist is able to squeeze both consumers and suppliers to the detriment of both.
It would be nice to find something better than an ad-revenue driven web, but I'm not sure this is it. We'll find out I guess...
Sure they are. I can attest that musicians will gladly publish their music even when no recompense is offered. Surely culinary artists are the same.
Nothing kills passion faster than your only audience being an AI crawler.
This is just disruption.
It’s also not disruption if your product relies on the output of the industry it kills. What will AI train on when it destroys the economics of sharing information with others?
I disagree, Google doesn't rely on a massive amount of recipe websites; they rely on Google and exploit it.
More importantly: These recipes exist from many sources, not just the website that exploited Google to be first in their rankings.
In this instance, the internet is more than top-ranking recipe websites.
Also, at some point even the ad-laden websites will die, and then Googles sources will be extinguished.
No because it's killing competition and becoming an even more obvious monopoly. Then at any occasion they have to choose between consumers and profits, they'll do what shareholders want and increase profits.
No. Temporarily it’s good for the consumer. Ultimately it is bad for the consumer, because as prices drop, so to does quality.
Same reason its not good to have a "company town" where 1 company is the major employer to 50% of the workers.
I'm not saying LLMs are worthless, but I'm saying if you had a magic browser add-on that simply stripped the BS out and showed you the relevant content, it would meet the use cases of the majority of people using AI (regarding search).
Said differently, Google are bringing you a solution to a problem they (largely) created.
I think it's a good tradeoff.
This discussion was broached originally when discussing whether or not search engines and aggregators had any compensation obligation in respect of news articles. This was a hot topic in the IP and policy circles for a few years.
When the Canadian government attempted to create a mechanism to compensate content creators for the scraped content, there was widespread outrage from tech circles, despite the same community agreeing, across extensive policy discussions, that action had to be taken to prevent this universal man-in-the-middle value capture by search engines.
I've had fairly extensive discussion with the individuals involved in the academic, policy and internal industry analysis of the issue. Watching industry agree to address the issue, then aggressively spend to shape public narratives in public was eye opening.
The recent shift into "AI is obviously going to hoover up all your data and there's nothing you can do now that the theft is laundered through an LLM" is just the latest example of the same trend of short sighted capital-over-everything decision making we've become used to in jurisdictions that have dysfunctional legislatures.
You’re free to block them, but the websites cloning your content won’t. So either way they’ll get the content they’re after.
Worse, when/if the time comes that LLMs source their claims they’ll refer you to the websites that cloned your content.
Let the big guys fight it out.
There is money in protecting the websites. If you host with OVH they are interested in you making money so you can pay them.
You are on top of Google search results or something? I think I am on some last page that no one will click like 2nd page of results.
The web won't die. It will just become even more commercialized than it already is. You might recognize most of the things listed.
Selling things: Restaurants, hotels, local businesses; Ads: more of the same
Work: a) Projects people collaborate on and need to share information b) Writing articles to boost a resume
And maybe sharing what people do, they do it for free, see social media.
AI summarization has already causes issues for sites like rtings where people are no longer visiting the site but still making use of the data presented there. Leading to rtings not getting enough traffic to continue to post their data.
It is an existential crisis for websites and when they go away it'll be an existential crisis for AI.
I may be strange and unusual, but I just have never cared about my Google ranking. I know this makes me out of the ordinary among site owners but I have been humming along fine.
This certainly will disrupt traffic but for some of my sites I honestly think this is a good thing. I want you to want to be there, not just stumble upon my site because you happen to hit the right search keyword. Plus if it gets bad, this does create a new opportunity for others with cross linking and search.
My target market is more technical then that so likely, nothing would change for me. Again, I recognize the impact of Google's dominance for some, but if the "attestation" isn't helpful and only hinders using services that people have come to rely on, there will be push back.
I also have been advocating for years for everyone in my circle to avoid using Chrome. A homogenized browser market is a risk, and Chrome is the new IE. I hope you are also a part of the effort to advocate for browser diversity.
They are already training people to click these away.
Do you depend on site visitors for making a living? That's what this is about.
I know that sites relying on ad income will and are being hurt tremendously by this effort on Google's part. However, if you are in the startup space and make money on services you offer, search should be one of several strategies you are deploying for user growth.
But that's why you're a nobody instead of a million dollar startup founder. You didn't play the game, so, you lost. I lost too.
Step 2, Google extinguishes the web and nobody has a reason to publish content, consumers lament but are trapped, Google has created a platform to serve content instead of links
Step 3 (or maybe 2a), Google is now monetizing their content machine
Step 4, Google offers people a way to contribute to the content machine, make some $$ per N views, whatever. People create content within the ecosystem
Step 5, Google is now the internet, more content is created overall, quality is lower overall perhaps, algorithmic echo chambers flourish even more than today, old heads on HN lament, everyone else just goes on living
There's no real mainstream way outside of youtube to make meaningful money with free video content.
But yes, interesting thought which does kind of make sense. The marketplace-ification of the textual web.
Imagine people would go to youtube and watch previews of your videos solely; or the preview is your video, but condensed and given preferential treatment.
I'm aware. And I'm curious how that will play out. Because same as with Youtube, historically Google Search gave the means and the discoverability to monetize producing valuable free content.
Youtube is directly dependent on people producing free content. If Youtube wouldn't pay its creators as well as it does, it would simply die.
Same with Google Search. Good content and good SEO gives the means for websites with free content to survive. Google usually takes a cut on ad placement on those pages with AdSense.
If Google Search now doesn't pay content creators as well as it used to, what will happen to free content on the web? It's bad for Google and it's bad for the creators.
We will see.
Still, it's all very centralized platforms, which historically isn't the case for all the monetized free content you usually get from google search (reviews, recipes, travel guides, converter sites etc. etc.)
Isn't Stack Exchange the emblematic case?
And here I thought denying ad revenue to websites was the morally superior way to navigate the web...
What about the stories of marketing managers who learned months after the fact that their credit card had expired and their google ad spend had ceased with no affect on traffic? Google isn't always an effective promotional vehicle.
I have seen this happen first hand at a Fortune 100. They didn't track shit.
this kills the entire internet vibe of the 90s, early 2k
FTFY: "couple of decades since has become". The vibes of passion-driven 1990s started to be overwhelmed by the din of money right when the Internet has become a major commerce venue, some time in early 2000s.
If your site is about your product, Google won't be able to serve the sign-up page from AI; the traffic would come your way. Same for a site that sell something: the traffic you're interested in would arrive at your checkout page.
Paid-content sites and ad-supported sites are screwed though, on top of their being screwed by archive.is and ad blockers.
That is not what "free mirror" is. Like, that is not the same thing at all.
(Torment Nexus rules apply here)
(It doesn't work for ad-funded writing, but while I have substantial sympathy there this has historically been an unpopular argument on HN)
This also could have been fine, it can bring back authenticity however for this to happen no one should be making money from it. Instead, only megacorps make money and they can just ignore your ideas and generate theirs. They control the distribution and the supply now.
More broadly, I love it when an author I trust in one area writes about other topics.
It's the news media that will suffer the most.
Websites may go back to being simply labors of love.
The situation may be even worse. Back in the labor of love era, at least webmasters could get feedback from readers. In the LLM era, readers may not even know that the site exists. Without feedback/community, the overall quality of those sites will decrease over time.
On the old internet, "know one knows that you're a dog."
On the new internet, "know one knows (for sure) that you're an AI."
In a mixed discussion consisting of both AI and human participants, AI could generate interesting discussions and recite novel-sounding statements for ongoing ones. It doesn't sound plausible to me that an avalanche of profit-driven LLMs would even post -- I would think they'd be the lurkers. My take would be that LLMs actively participating, depending on the content they post, are also labors of love by some nerd out there trying to make the best LLM.
Drain all the profit motives and rent-seeking from the web, and even something like AI joining discussions doesn't sound scary.
ChatGPT/Claude does this today. I barely click or care for the source when they already have me the info I wanted.
My speculation is all information worth anything is going to be behind some kind of wall.
Maybe I'm just #builtdifferent, but I click these a lot. Especially if I'm trying to research or make a decision on something, I want the actual source and not the potentially-fudged summary.
Not to mention the hallucinations
Similarly, if I use Gemini uses a website for an answer, it should pay something to those sites for the information it gathered. Sites would need to sign up to earn via Google, and I'd imagine there would be a certain threshold to cross to make it worth cutting checks... but that would make all these AI search tools feel much less scummy while providing site owners an incentive to keep sharing information on the internet.
Where a model like this would get messy is with sites like reddit. It's a very popular source for AI search, but the value comes from the users, not the platform itself.
The problem with all this AI/llm stuff is that end users doesn't even know your tiny site with a lot of useful information exists at all.
This depends on implementation. I primarily use Kagi for any LLM stuff. I cites pretty much everything and links out to the source. I regularly use this for search. The normal search results may not have what I need, but a line in the AI results sounds better and I click through to the source to get more context.
I find clicking through to the source is important, as I've often seen the AI get it wrong. The page has what I need on it, but the AI grabbed the wrong thing and got it backward. I'm probably in the minority, I'm guessing most people don't use LLMs like this.
- in the past Google was the entry point for everything, I was opening every single site at the top of my search and navigate through it. E.g.: if the site was a Reddit or HN post I was reading comments, following links, ecc
- today I'm using google 1 in 3 times and I mostly read the "AI Overview" section. The other 2 out of 3 I go directly into chatgtp, claude or gemini and rarely follow any links.
[0] But I see the same pattern basically in all of my colleagues, friends and relatives.
If I'm gonna lose my job, at least give me that in return
In that case, the consequence will be that people will stop having webs. It is already happening with personal and niche sites.
As far as I know, you don't have a choice. They have no obligation to respect your wishes, and LLMs are legally allowed to scrape & republish your content.
I have no obligation to not send all scraper-looking traffic to a black hole full of zip bombs.
Showing porn to a minor is not legal either.
Disclaimer: his website is for hosting malware for "testing" purposes. Testing how well AI can't deal with it.
But fine. How about I just...don't respond to those requests at all. I have no obligation to send them data period.
If instead the purpose of your website is to manipulate users for financial gain (for instance by showing media attempting to manipulate their purchasing decisions, after receiving a bribe from a vendor), and the information is just a way to lure users, then maybe this malicious business model will finally be no longer possible.
On the other hand, Google played a big role in creating this problem in the first place. The search results have trended downwards towards this kind of SEO slop for the last 10 years and Google has been unable (or unwilling) to fix it. Plus, the AI results Google shows are not free from commercial influence and will probably only get worse in this regard. Except now this money will flow to Google instead of $random_internet_spammer. I don't know if that's any better.
The idea that Google won't eventually "manipulate users for financial gain" with Gemini is comically naive. That's how they're going to make money from this thing.
The counter argument is that sites are becoming more AI slop or may intentionally provide poison they don’t want to train on. There may be a cut off date after which training must be carefully curated; and the main body of data has already been collected.
Sites may still get traffic from agents searching for current information. Maybe even the resurgence of RSS? One can dream.
Mechanisms might exist to make you think you have one, the same way copywrite should prevent millions of books being gobbled up by TheZuck but ultimately do you really have a choice?
Rules and laws don't exists for you.
this has been done before, quite often, but toward ends morally askew.
https://developers.google.com/crawling/docs/crawlers-fetcher...
Site traffic
My income isn’t ads, just getting a cut of the sale on the complex products I help you buy. Even that sort of curation takes time and effort.
Even for all the things I do for free without any revenue whatsoever - most of it, really - I do want to feel some recognition. I don’t want the interaction to be mediated by an advertising company.
Mention
Google has always crawled your site and been an arse! Now you get to decide whether they are hallucinating!
You can drop pointers on Masto and other socials to your sites - that has not changed.
Do we need something else? ie you drop a link to somewhere else.
Making the information available that you put up your site for?