Hacker News Reader

Comment by alightsoul | Hacker News Reader

I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving them money

There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages

View on HN

> There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages

Where do you get stats on this info? Not refuting your claims but it's interesting data. I would've thought a wider gap in all biz to active websites since some countries run nearly all their biz on whatsapp/telegram/wechat.

I remember 300 million is the number of businesses on Google maps. I forgot how I got that number though. I didn't crawl Google maps. A lot or even most of those businesses run via chat often have no physical presence besides the houses of their owners so houses on Google maps are not counted. There are a lot of businesses on shopping malls and B2B businesses that aren't on Google maps often because they don't know they can add themselves to Google maps.

I did count the number of ICANN domains on their zone files on czds and there are a few services selling pre compiled domain lists that do crawling for marketers (it is forbidden by ICANN to use zone files for marketing) I don't remember how I got the active website count of 100 to 150 million, maybe I interpolated by checking some ICANN domains or there's a domain list service that does it. Domain list services usually say how many domains are active regardless of whether they have a website or not

Of course parked domains add up to the active website count even though they have no relevant content

Both IA and CCF are private foundations and accept donations.

Reply on news.ycombinator.com