Google crawl was originally "seeded" by starting with the human curated DMOZ directory. Eventually they stop doing that and have gone fully non-human.