> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found.

I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source.

I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...