Don't have any datasets but http://news.ycombinator.com/robots.txt is your guideline.