Hacker News Reader

> There is no reason to believe you are getting good information this way.

Spot checks checked out. And it was a perfectly fine way to do it.

You are correct that Tor exit nodes often get special handling (at the moment, including by Cloudflare, and by Google Recaptcha). And the idea of poisoning of data is starting to propagate, due do anti-AI-scraper sentiment.

View on HN

Poisoning the data has been around for at least a decade now. I used to work for a very large pricing analytics company that would track product pricing for Fortune 500 manufacturers.

We found on several occasions that some shady retailers would find the CIDR of the manufacturers corp networks and comply with the MAP policies on pricing if traffic came from them. Then when our bot went through with obviously generic AWS / Proxy ips we would see a much lower price that broke their agreement. That one was a fun realization for the manufacturers as to the level of shadiness some retailers would go through.

Reply on news.ycombinator.com