Hacker News Reader

It starts with seed URLs and fans out from there

Back in 1996, this could be done by hand

thanks! I imagined it will be something like that. Maybe Google used yahoo index back then as seed?

Yes. I think, there used be a directory of a links published, like an index.