Just curious. Wasn't Yahoo using Google and Bing for search? Is this crawler being used at Yahoo internally?
I have no idea if this is what was used here but I know this is an example where I'm sure they didn't out source the search work to Google.
I have no idea if this is still the case.
They did things like Yahoo webpage hyperlink connectivity graph, and tons of other research.