Web Crawl Analysis: Unveiling 465K Shopify Domains from 3TB of Data

alistechtales.substack.com

7 points·ghafouri··10 comments

Dived into Common Crawl’s 45 billion URLs, processing 3TB of data using Python's aiohttp and multiprocessing, Kaggle, and MinIO. Resolved domains with Google’s and Cloudflare’s DNS over HTTPS, uncovering over 465,000 Shopify domains. Discover the code and data insights here!

I don’t get it.

Why is scraping Shopify domains a goal / interesting?

Maybe to use known vulnerabilities and take over sites?

Came here to ask the same thing. Also, if they are all on the same IP, what about reverse dns lookups?

With reverse DNS, I only get a few results, not hundreds of thousands of domains. So, I took this approach.

Generate a list of leads for Shopify related services or Shopify competing product.

I used to work on a Shopify SaaS project, but it was canceled. I'm thinking of sharing some of my work from that project.

Why pound Web services of Google or CloudFlare for DNS lookups, instead of using the DNS protocol?

Were DNS servers balking at the continuous 100 simultaneous requests to resolve much of the Internet?

I achieved 1000 DNS resolves per second with DOH, and my attempts on DNS servers are fewer.

Have you thought about productizing this as a better reverse DNS lookup? It feels like you're 90% of the way there.

Reply on news.ycombinator.com