Ask HN: how much does it cost to crawl 1M sites?

1 pointvisox4 comments

What is the cheapest api or other solution ? How much more does it cost to crawl sites with javascript ?

thx

View on HN

I think i made an unclear/wrong question. I wish to crawl about 1M pages (rather then sites)

so 1M http get requests.

I was hopping there would be some solution for 100$ lets say

$0. Crawl from your computer at home.

Depends on which sites. Some of my websites have 200K pages. Some have just three.

Also depends on what you want to do when you crawl them. I guess if it's a million sites it is going to be median distribution of number of pages. But I don't know that information is available anywhere. I think the http archive used to tell you the number of pages on sites, but now it just tells you page sizes.

https://httparchive.org/reports/state-of-the-web?start=lates...

of course page sizes might also be important dependent on how you crawl and what you do with the data.

Reply on news.ycombinator.com