I don’t pretend to know all of the motivations behind the policy moves.
That said, I’ve got some experience with scraping; got sued by LinkedIn in 2014. We were using AWS Spot Instances to hit it hard for very little money. It was not uncommon to accidentally take large services down.
Scrapers can and do add very significant load. We also scraped Twitter back in the day as well.
Twitter was very good at this, and their new-found inability is a glaring sign that their engineering is slipping.
You simply setup API deals with those who you want to have your data, those that benefit your business, aka Google etc…
Then you close everything else up. This saves cost and complexity and real users, the target of your advertisers, don’t even notice.
This isn’t a sign that engineering is slipping.
It’s a sign that a in a company which struggles to make money, someone is paying attention and trying new things to fix the money problem.
What I said is that they must handle the problem transparently to their valuable users. That includes (requires, usually) targeted techniques to block high-volume scraping.