This is really interesting. Thank you for the explanation.

I'd love to hear about how you crawl and store the data. Doesn't shopify block you as you are hitting there servers soo much for all the site?