Google Banishes Nicholas Carr From The Index
roughtype.com
roughtype.com
Maybe it's just hidden spam links? In which case, index-removal seems an extreme way to get the owner's attention.
I realize neither this comment nor the one on Nick's blog from my Google colleague Greg are authenticated, so you'll just have to trust me that I'm me :).
We do often remove entire domains from our index when there's evidence of hacking and/or malware... not to be mean, but rather to protect both visitors to that site and the site owner (from angry visitors). We can't be sure that the hacked links aren't or won't soon become particularly malicious (e.g., phishing sites), nor can we be certain that the server hasn't been or won't soon be more substantially compromised (with drive-by malware installs, etc.), so we fall on the side of caution.
We do make a good faith effort to contact the webmasters affected... - Through e-mail addresses we believe to belong to that webmaster. - Via our free Webmaster Tools service, where webmasters can sign up and opt to get e-mails from Google on just such occasions as this.
For more info, you can check out our most recent blog post on this topic: http://googlewebmastercentral.blogspot.com/2009/02/best-prac...
Hope this helps! Believe me, we want good content back in our index as soon as possible, too :)
The "could get worse" standard is a slippery slope, unless the evidence suggests a very specific serious compromise or foreign active content. (Exiling a site because a few evil links appear in its comment threads, for example, would put almost every site at risk of removal. Indeed, that standard risks gaming by a site's rivals, using comment forms rather than 'hacks'.)
Also, looking at Carr's last comment, even though he's removed the bad links, he doesn't know what compromise allowed them to appear. So his site is still in the "can't be sure... won't soon become particularly malicious... [or] substantially compromised" category that justified its initial removal.
Your deep and sensitive checks for problems are definitely a public service... as long as the standards are clear and communications effective. Maybe notification should occur via a site's own comment functionality in addition to email? No details for third parties to exploit -- just a "please check Google Webmaster Tools" note.
I definitely hear you on the slippery-slope issue, and perhaps I could have worded my initial note more carefully. We have reasonably conservative heuristics in place to assess whether a site has been actually hacked vs. just someone adding, say, a link to a nasty site in a comment. Also, in our experience, once a site or server's been compromised, there's a good chance that, well, where there's smoke, there's fire.
Regarding giving more information... I believe we do generally show specific examples re: hacking and malware via our Webmaster Tools, though I can't say for sure what we presented in Nick's case. Generally, we've seen that webmasters are able to catch the root vulnerability (via their own unpatched software or via their server), which prevents the same problem from recurring. (and speaking of unpatched software... from my own spot checks, that seems to be one of the most major culprits!)
Lastly, re: your suggestion about notification on the site's comment functionality itself... that's a fascinating idea! I think that it'd be a bit challenging to implement, given the diverse array of registration requirements, captchas, etc. (our bots are smart, but not as smart as humans!), but I know that our team is looking into more ways that we can let webmasters know of issues more effectively. Thanks for the suggestion!
Brings up all sorts of URL's that contain the string roughtype.com but are not the site itself.
It's frightening, especially since we have no way of knowing what else is being filtered out and for whom.
I would tend to give the big G the benefit of the doubt; it's a mistake or he got on the wrong side of the algorithm, or his site was infected with malware. (that last one does happen see http://blog.oddhead.com/2009/06/22/un-hacking-my-blog/ )
But still, it is the reputational equivalent of logging in to your brokerage account to find out it's empty.
Do you have any data?
We would all be trying meta-search engines that compared different result sets. We would be back in the late '90's trying to figure out which search engines were reliable and honest. (anyone remember Dogpile?)
Except that now it's all incumbent monopolies and high barriers to entry.
If nothing else this should give a boost to small outfits like DuckDuckGo http://duckduckgo.com/?q=site%3Aroughtype.com
As Matt explains, he was re-added to the index with no explanation. Perhaps Nick Carr will have a similar redemption.