390 karma · joined September 21, 2014
2013-2021 is 8 years. That's an average of 5,000 per year or roughly 100 per week.
Wild.
Even so- where we do draw the line on safety? When does this cause an accident because someone didn't notice a flat tire and drives on the freeway with it?
Looking at you Applebee's.
The irony for us as a provider is that it's one of our customers (party A) paying a third party to scrape data from another one of our customers (party B) which in turn affects the performance of party A's site. We've started blocking these third parties and directing them to paid APIs that we offer.
We chose to switch to the JS challenge screen as it requires no human interaction. We now block 75% (estimated to the best of our knowledge) of bot traffic but some customers are livid over the challenge screen.
edit: Cloudflare has responded that "[they] are asking your question internally."
Since Feb 2021 we've seen a substantial increase in scraping of our customers' inventory to the point we now have more bot traffic than human traffic. Annoyingly, only 5-10 requests come from a single IP so we've had to resort to always challenging requests that come from certain ASNs (typically those owned by datacenters).
This type of project frustrates me because it knowlingly goes against a site's desired bot action (via robots.txt).
Brave's Sync v2 works decently well.