Cool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?
Does "Built With" provide that data? How accurate do you think it might be?
# / <//_\
# ,:
# ,' |
# / :
# --' /
# \/ />/
# /
# ,: # ,' | # / : # --' / # \/ />/ # /
Ref: https://html.spec.whatwg.org/multipage/parsing.html https://developer.mozilla.org/en-US/docs/Web/CSS/white-space...
The robots.txt does not exclusively list what not to scrape.
It provides information on which parts are allowed and wich are not (disallowed).
It also provides sitemaps for crawlers as a starting point with more information (eg. which sites are available and how often are they updated, etc.)
Shopify sites also have shop-name.com/products.json which has URLs that point to cdn.shopify.com