In this case, because there's a vastly more efficient economic path for the adversaries (cloning).
They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
(Also, blocking a whole country is likely not what you do, but you probably know that).
Oh, and it also only takes a few minutes to reconfigure everything and invalidate those years of legal and investigatory work.
This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.
Developers and compute are interchangeable now.
It's more likely that they're trying to scrape the entire internet, and cgit is just unfortunate enough to have autogenerated pages, where each page seems to contain text entirely distinct from its neighbors. This probably bypasses the scrapers' heuristics, which normally prevent them from getting trapped in a maze of twisty little autogenerated pages, all alike.
The way you normally deal with datasets at that scale is to move postprocessing (particularly expensive postprocessing) as late in the pipeline as possible.