And this is why scraping is under technical and legal attack. To maintain their panopticon, where they see all, while you get only a pinhole view of, in this case, the economy.
Contrary to what many people thing, Facebook, Google, and other companies do not sell your data! They are ad companies that use their private data to target ads, but they aren’t selling it to 3rd parties likely many people have been led to believe. This has created a market opening for nefarious companies to scrape these websites, compile the data, and sell it to everyone from police to governments to spammers.
I’m not saying there’s a right answer to regulation, but it’s incorrect to say that scrapers are always the good actors championing the public’s best interests.
I don't want Google or Facebook to have my data either, yet they've done their best to gather everyone's with various trackers, not just data posted voluntarily to their platforms. They don't have to sell it - they are the threat too.
I liked how the thread observed that competitor supermarkets were rapidly converging prices on adjustments.
And assumed this meant collusion.
Why collude when you can just hire grey-market high frequency scrapers to detect your competitors' repricing and trigger your own?
Probably isn't something a company with potential monopoly concerns (read: Walmart et al.) would want be be overly open about.
What the thread actually said is this:
> My guess was: tacit collusion, meaning, oligopolic price coordination without explicit coordination.
Which is what you described.
this does not excuse the pervasive tracking that occurs by "we collect for internal purposes only" vs "we sell to anyone". an invasion of privacy is still an invasion of privacy whether they do it for their own fetish purposes or distributing it to the public.