Some other notes: 1) you're not checking subdomains like blog.company.com or paths like company.com/blog 2) if you use something like zgrab you can do 1M site crawl in a couple of hours. Consider checking it out.