What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?
The cat and mouse game happens at every level, not just the DOM/browser-detection level.
As someone who routinely works behind proxies, I can sympathize strongly with this statement:
"The one thing that I was really trying to get across in writing that is that blocking site visitors based on browser fingerprinting is an extremely user-hostile practice."
That said if you try to scale this up beyond what a reasonable, normal user world do in one sitting, you are bound to stand out.
Although that said, I find that I trigger such rate-limiting mechanisms already as a human just when searching Google as a human being and clicking through every last search result page.
Sure, but that's easily mitigated by running multiple scrapers as different users.. You don't need to get all the data from a single scrape.
Recap [1] does this to extract PACER court documents that are public domain, but access is restricted due to draconian public policy.
I think Google claims to try to detect humans by parsing out their mouse movements and scroll events.
At this point it would be easier for me to write an alternative frontend to Google search (or just use duckduckgo), but it was be amusing to think that I might evade this by writing a script to simulate mouse movements to appear less robotic.
In my experience this occurs when you are either doing this too much, or you are not accepting their cookies when logged in. (I don't recall the behavior when logged out.)
Actually, for whatever reason, this also often seems to cause reCapcha to essentially hellban me and just keep asking for me to solve an endless series of capchas. :/