It's a combination of things.

As far as I understand it, it's fingerprinting + IP reputation + cross-site IP rate limiting + raising the cost of a request by an order of magnitude because of the LLM call.

The last ingredient is what we forget about. $0.001 or whatever might not seem like much, but there's a difference between "a million requests a day costs you $5 in AWS bandwidth" and "a million requests a day costs you $1000 in VLLMs, $500 in premium bandwidth from residential proxies, + constant toil to update your scraper to keep up with what Google is doing."

(Figures illustrative).