Hacker News Reader

> and the data that I’ve experimented with from 2014 seemed high quality

That's because it's from the blekko search engine.

Sounds like blekko had a larger impact on the early urls than I thought. Out of curiosity, do you remember how large blekko’s index was at it’s peak?

The largest index we had was 4 billion, which is tiny. Our crawl frontier was much larger.