It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
(Also, blocking a whole country is likely not what you do, but you probably know that).
Oh, and it also only takes a few minutes to reconfigure everything and invalidate those years of legal and investigatory work.
This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.
Developers and compute are interchangeable now.