So, ideally for you, SEs would have to be opt-in and do forceful revenue sharing with page creators?
You realize there is no way something like google could have been created in your proposed environment?
You realize there is no way something like google could have been created in your proposed environment?
SEs are already becoming opt-in for some sites. Extranets are a great example. Some people are already building up robots.txt files that either exclude by default, or they have long lists of blacklists that they add to whenever they notice a new spider. For example, look at http://news.ycombinator.com/robots.txt. It disallows /reply? and /user?, which is basically all the content on the site. I expect that a large percentage websites will become more like opt-in instead of opt-out, especially as the number of robots increases--eventually, there will be so many robots spidering sites so frequently that bandwidth caps will be exhausted just from the robots.
I don't believe in "forceful" anything. But, I do believe that large, popular sites will make by making (exclusive) revenue sharing deals with search engines in the future. And, robots.txt can be used to advance some social policies too--for example, you could cut Yahoo off if you don't like its human rights record. Copyright is what will enable those kinds of things to happen.
I also think that Google's domination of search will come to an end and there will be more incentive for publishers to listen to offers that other search engines make. Google is probably not going to offer revenue sharing (from ads off SERPs) first, but I bet somebody else will. By this, I mean that the search engine will share revenue from clicks with the sites that appear on the SERPs that the clicked ad was on. It is the naturally evolution of SE advertising, and it is the only thing that will make SE's increasingly agressive use of publishers' and authors' content bearable for some people.
User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply?
user is not disallowed anywhere, vote? and reply? and x? are, but they are not content, they are actions, basically all of the content of this site is indexed.
Your idea of site owners being selective about SE makes no sense to me. Would users then have to use 5 or 6 different search engines to accommodate all possible webmaster preferences?
I think that SE revenue sharing will happen. Only the SEs that share revenue will survive--a SE that doesn't share revenue will get locked out of the most valuable content and it will be forced out of business.