Is that true with how they trained Gemini? Doesn't everyone with a foundational model scrape the web relentlessly without regard for robots.txt?
Almost everybody wants to appear in search, so disallowing the entirety of Google is far more costly than E.G. disallowing Openai, who even differentiates between content scraped for training and content accessed to respond to a user request.
Like if you give a friend a key to your house so they can check on your plants when you're out of town but they throw a rager and trash the place.
That was not a phrase I expected to read on Hacker News! Haven't heard it since I was about 13. I always assumed it was a Scottish phrase.
But yes, I am dating myself with that phrase...