Greg Linden's (MSFT) comments on a recent Google paper on this:

http://glinden.blogspot.com/2009/01/how-google-crawls-deep-w...