The thing is that most ecommerce websites of note generate shopping feeds in easily machine readable formats (JSON, XML) for Google Shopping, Facebook and the like. These feeds also go down to SKU level. The URL might not be advertised but it won't be blocked or protected with a user/password API key.
If buying a T Shirt, the product page might list all sizes and all colours only showing a master 'variant' SKU (that is not a real SKU) and the backend might then add the actual size/colour specific SKU to the basket, of which there could be twenty on the 'variant' product page.
Meanwhile the product feed will list every SKU variant, complete with latest pricing and other pertinent information, e.g. barcode, product image etc.
I am sure that most retailers would prefer to just point the scraping party to the feed rather than have them grind the site to a crawl with multi-threaded crawlers hiding behind proxies. So that is how these scrapers can be 'accommodated'.
The sitemaps that go with the ecommerce game are also pretty reliable, these are high up the SEO checklist and will say when products were last updated.
Then there are rich snippets - or whatever they are called now. The trend in these is to have some JSON-LD attached to the page in some format GoogleBot likes. Not hard to ingest.
Sites that don't have their act together for Google Shopping and SEO really are not worth scraping, they will never make it to the Google top 100 search results unless they are selling something that nobody else sells, e.g. 'Tibetan Monkey Stones' where you probably don't need to compete.
To me it sounds like these scraping concerns just need to pay a bit extra for ecommerce developers to show them how the 'puzzle was made' and to stop abusing people's business websites that are not built to be scraped on a daily basis by some random third party on the other side of the globe.
Also the plain old telephone helps. If your brand owning ecommerce team get a call from an interested party saying that they would really like to get a list of their products for their comparison/whatever site then they just might say yes, here is the URL for the feed, oh and here is the one for locale_en_xy. But people would prefer to hack away at some hacky spider rather than just pick up the phone and ask.
So I would think most scraping services assume they will be refused when doing such requests so they never bother.
The only valid reason for starting a large scraping operation is because you can argue that permission would be granted anyway.
Additionally, in my local market the owners of e-commerce websites are extremely narrow-minded and have zero tech education so all they will ever hear from you is "I want to steal that guy's data" which is of course not true at all. But try and argue with a 50-year old guy with the mindset of a feudal master who never truly worked in their life but want to control how everybody around them works.
If the survival of my business was at stake, I would just scrape one page every 3 or so seconds as a reasonable compromise. In fact I have done so for my amateur scraping experiments, although there the timeout was even steeper -- 10 seconds per page.
As demeaning and offensive many people would find that statement to be, I still found it to be the sad reality most of the time.
Plus my local community is much smaller and I would not want vengeful businessmen who understand NOTHING from what I am trying to achieve, to actively sabotage me. They can easily call my ISP and deny me service, for example.
So I opted for ethical scraping without asking questions. Seems to be the best working compromise.
Thanks for sharing your experience. Let's bathe in the confirmation bias it dips us in. :D
I did find it surprising that this article has a whole section on "Challenge 4: Anti-Bot Countermeasures" (and how to bypass them) but doesn't mention giving any consideration as to whether this is a reasonable way to behave.