The robots.txt is only part of the problem, simply overloading sites is another. Maybe you don't really mind that your site is getting scraped, but you do mind that it crashes due to poorly written scrapers.
It was always understood that your scrappers should limit their load to something that a site can handle. Google has always excelled at this, I've never experienced the GoogleBot being an issue, nor the BingBot. The developers who work at the AI companies however are less talented and less careering. If their bot crashes a site, meeeh, try again later.
I don't think they DDoS sites deliberately, they just don't have the skills to fix their shitty scrapper.
If we want the problem to go away, ask AWS, Azure, GCP, Alibaba Cloud and others for a way to report bad behaviour. Make their hosting providers take action.
It doesn't matter whether they have the skills or not, because they don't care.
As long as they can siphon up enough data to increase their revenue or stock price by 0.001¢, they're perfectly content to leave small independent websites shattered and unusable under the weight of their aggressive, unrelenting scraper bots.
I'm firmly of the opinion that they should all be considered hostile, predatory companies, regardless of what supposed "value" they bring.
Train your river-guzzling models with data that's paid for or given voluntarily and then we can talk about your "value".