wrt robots.txt, what is the incentive for site owners to prevent duplicate submissions? As a site owner, wouldn't I be happy if both http://www.myadsenseblog.com and http://myadsenseblog.com get submitted? 2x15 minutes of fame!
* votes aren't split across multiple copies: a few votes here, a few there, and maybe you could have made the front page
* users are less pissed off (looking at you, nytimes and bbc)