Hacker News Reader

If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.

You say that like it doesn't happen all the time. googlebot included.