Hacker News Reader

Comment by altdataseller | Hacker News Reader

All these top million lists are very good at telling you the top most 10K-50K sites on the web. After that, you're going into 'crapshoot' land, where the 500,000th most popular site is very likely to be a site that got some traffic a long time ago, but now isn't even up.

So I would take this data with a grain of salt. You're better off just analyzing the top 100K sites on these lists.

View on HN

> where the 500,000th most popular site is very likely to be a site that got some traffic a long time ago, but now isn't even up.

That's literally the phenomenon the article is describing.

Ok let me reword it differently: the 500,000th most popular site on these lists most likely isnt the 500,000th most visited and it might not even be in the top 5 million. These data sources are so bad at capturing popularity after 50k sites or so simply because they dont have enough data

I haven't tested this, but the "Cisco Umbrella 1 Million" is generated daily from DNS request made to the Cisco Umbrella DNS service. That seems to be a very good and recent dataset.

It does count more than just visiting websites though. If all Windows computers query the IP of microsoft.com once a day that'll move them up quite a bit. And things in their top 10 like googleapis.com and doubleclick.net are obviously not visited directly.

So while it is quite a reliable and recent dataset, it is not a good test of popularity.

How are people determining the "top" sites? We do some of this at work and we pay SimilarWeb a giant sum of money, are people able to find site traffic in inexpensive ways which allow for these analyses?

Reply on news.ycombinator.com