Gemini is currently embarrassingly bad given it came from the shop that:
1. invented the Transformer architecture
2. has (one of) the largest compute clusters on the planet
3. can scrape every website thanks to a long-standing whitelist
This only remains true as long as website operators think that Google Search is useful as a driver of traffic. In tech circles Google Search is already considered a flaming dumpster heap, so let's take bets on when that sentiment percolates out into the mainstream.
How many websites back then would be youtube channels, podcasts or social media accounts if they had existed back then?
Nowadays most sites survive via traffic from google, if it goes away then most of those sites go away as well.
1. They were a major site that was an initial starting point for traffic
2. Search engines pointed to them and people could locate them.
---
That was all a long time ago. Now people tend to go to a few 'all in one sites'. Google, reddit, '$big social media'. Other than Google most of those places optimize you to stay on that particular site rather than go to other people's content. The 'web' was a web of interconnectedness. Now it's more like a singularity. Once you pass the event horizon of their domain you can never escape again.
Websites won’t be blocking the search engine crawlers until they stop sending back traffic, even if they’re sending back less and less traffic