User-agent: *
Disallow: /
Asinine
We try to respond to every single e-mail we get; not sure how yours was missed. I apologize.
We've talked about our policy on re-publishing comments in a few places:
- http://www.markevanstech.com/2008/08/30/five-questions-withb...
- http://blog.backtype.com/2008/08/on-republishing-comments/
- etc
In summary: what BackType is doing is very similar to other search engines. We make every effort to provide content publishers with familiar methods to restrict BackType from indexing their site's content: http://www.backtype.com/faq#q14
We're trying to be as transparent as possible. We attribute comments to their authors so they can be rewarded for the thoughts and insights they share in comments across the web. Additionally, BackType is already becoming a recognized source of traffic for publishers and we want that to continue.
When I comment on TechCrunch or 37signals, I don't give them the copyright to my comment. I (implicitly) give them a license to republish the comment on their own website. However, I've never given Backtype a license to republish my comments on its website. Even if TechCrunch or 37signals chooses not to opt out of your system, you still don't have any legal rights to republish the comments.
From the commenter's standpoint, does it make sense to let you republish my content? I think the answer is emphatically "no." If I republish my own comments myself then I can also monetize them myself. If I build a WordPress plug-in to assist others in doing this, and/or if I build an OpenID extension or robots.txt extension for making comment streams more discoverable, then I can help form a user expectation of "If I want to see all the comments by John Doe, I can find them on John Doe's website." Then all the John Does can decide on their own how they monetize their own content on their own website (e.g. AdSense).
Consequently, really you are in direct competition with the people whose content you are re-publishing. AFAICT, you are saying "we're going to profit from your content without compensating you, regardless of the copyright, and there's no way that you can opt out except by refusing to publish new content." That sucks.
In the posts you linked to, you are comparing yourself to search engines like Google. The difference between your company and Google is that your company republishes the content in full, but Google provides just a small clip of the content. Google has a fair claim that what they are doing is fair use. Their limited excerpting of content is also what allows them to use an opt-out model instead of an opt-in model. That is also what makes it is easy for content creators to profit from Google's indexing of their content--Google users have to click through to the original site to see the content. You are just re-publishing the content in aggregate form.
I'm also curious about whether you think the DCMA safe harbor provisions apply to you when you are republishing others' content. As far as I can tell, they cannot, since you've really just built an automated plagiarism system. That is, there is no user-submitted content on your site; all the content published on your site is published by you yourself.
Anyway, if you really didn't get the message I sent your company, it boils down to this: Please remove everything I've written from your website, or come up with a means of fairly compensating me. Thank you.
But, it doesn't matter how I feel about Google because Google isn't republishing content beyond what is allowed by fair use. So, even if I hated what they were doing, I don't have any legal justification for asking them to stop.
Individual comments make up a very small percentage of a page's content, so what we're doing is more similar to Google than you suggest (they also index your comments -- search for them). We display excerpts of pages; we attribute them liberally. A search result on Google leads you to the full page of content; a search result on BackType leads you to a full page of content. We index content for purposes of discovery i.e. search.
== Fair use
Also, it's pretty obvious that we're not in competition with content publishers, and we provide a very effective way to prohibit BackType from indexing comments.
I'm not talking about the rights of the weblog owners. I'm talking about the rights of the commenters on those weblogs. If you search for my comments on backtype, you get the full text of each comment. That goes far beyond fair use. You can search for any fragment of any of my comments on Google and it won't show you the whole comment, it only shows a (fair use) excerpt.
To be extra clear, you "excerpt" pages but you are not excerpting the individually-copyrighted comments. It is analogous to a website that republishes a book a page at a time and then claims that each page is a fair-use excerpt of the book.
Anyway, I don't see how you can say that your assertion is obvious when I've already explained the ways you compete with content publishers and content authors. Besides that, as the Plagiarism Today piece mentions, you compete in SERP placement since you let search engines index the content you plagiarize.
So what are your goals here? Are you trying to monetize your writing, or are you trying to prevent others from doing so?
Right now I'd rather not have my comments aggregated at all, anywhere. But, if I change my mind, I want to be in control of the aggregation--who does it, what content is aggregated, what details are disclosed, and how much I am compensated for it.
I think eventually all search engines will have to do revenue sharing with content producers. Because, while search engines provide a useful service (even Backtype might be useful if it had a bigger index), they cannot exist without content to index. Plus, search engines are increasingly competing with content publishers (e.g. Google Knol, everything on Yahoo).
So, really what I am opposed to here is the opt-out (instead of opt-in) nature of the republication. I would be less opposed to it if there was an effective and convenient way to opt out of it. It seems right now the only ways to opt out are to file DMCA complaints and/or lobby websites to exclude the spam bloggers' robots in robots.txt.
You realize there is no way something like google could have been created in your proposed environment?
SEs are already becoming opt-in for some sites. Extranets are a great example. Some people are already building up robots.txt files that either exclude by default, or they have long lists of blacklists that they add to whenever they notice a new spider. For example, look at http://news.ycombinator.com/robots.txt. It disallows /reply? and /user?, which is basically all the content on the site. I expect that a large percentage websites will become more like opt-in instead of opt-out, especially as the number of robots increases--eventually, there will be so many robots spidering sites so frequently that bandwidth caps will be exhausted just from the robots.
I don't believe in "forceful" anything. But, I do believe that large, popular sites will make by making (exclusive) revenue sharing deals with search engines in the future. And, robots.txt can be used to advance some social policies too--for example, you could cut Yahoo off if you don't like its human rights record. Copyright is what will enable those kinds of things to happen.
I also think that Google's domination of search will come to an end and there will be more incentive for publishers to listen to offers that other search engines make. Google is probably not going to offer revenue sharing (from ads off SERPs) first, but I bet somebody else will. By this, I mean that the search engine will share revenue from clicks with the sites that appear on the SERPs that the clicked ad was on. It is the naturally evolution of SE advertising, and it is the only thing that will make SE's increasingly agressive use of publishers' and authors' content bearable for some people.
User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply?
user is not disallowed anywhere, vote? and reply? and x? are, but they are not content, they are actions, basically all of the content of this site is indexed.
Your idea of site owners being selective about SE makes no sense to me. Would users then have to use 5 or 6 different search engines to accommodate all possible webmaster preferences?
I think that SE revenue sharing will happen. Only the SEs that share revenue will survive--a SE that doesn't share revenue will get locked out of the most valuable content and it will be forced out of business.
10 * $0 = $0.