It seems obvious to me that Murdoch et al. want to stay in the Web and be paid rather than de-index and disappear. Given that, these "just use robots.txt" articles seem just as disingenuous as the original statements they're mocking.
"go to Google News, or type a newsy topic like "Obama wins Nobel" into Google's search box. What do you get? Headlines and very brief teasers linking to news stories from news sites. If you click on them, you are taken to that news site, where you can read the story, which is surrounded by that site's ads. What, exactly, did Google steal in this scenario?"
What exactly did Google steal? Very exactly, Google steal "Headlines and very brief teasers". From the perspective of copyright law, yes, it can be classified as quoting, so they can't sue google, but those "headlines and teasers" do carry news value and in some cases took large amount of resources to produce. Now, Google took it and provided link as a payoff. Is that payoff adequate? Murdoch and many news producers don't think so.
Of course, teoretically you can use robots.txt. But in reality you can't, since your competition is also indexed, so you have to play by Google rules, even if you believe they didn't pay you enough for the value they are taking.
Do you have much sympathy for me?
Why, then, do you have much sympathy for Rupert Murdoch and his billion-dollar corporation?
It's an analogy, and like all analogies, it's imperfect. But it's perhaps more apt than you'd expect. You might argue that I could easily quit and get a job at, say, Yelp or Facebook. But News Corp could easily block Google and form a partnership with Yahoo or Microsoft - heck, there's rumors that they are doing just that. Or you could say that I should quit and form my own business - well, News Corp could easily go hire a crack team of 100 programmers to build their own indexing engine. Except they probably won't find the good ones, since everything indicates that Google is a better place for a computer scientist to work than News Corp.
You might also argue that I'm already paid for my labor. Well, News Corp is already paid for the their headlines and their brief teasers: they're paid when people visit their page and advertisers put up money for that privilege. They just want to be paid more. It's not all that different from me wanting a higher salary because I generate far more value to the company than I cost.
My point isn't that you'll get fair treatment (you won't), it's that life's not fair. It sucks to be an employee, it sucks to be a small business owner, and it particularly sucks to be one of the above when all your counterparties are giant monopolies. But as a society, our response to most people in these positions is "Suck it up and deal - you knew what you were getting into when you took the job." News Corp has been on the winning side of the "We're the only game in town" bargain a lot - they just finally met someone who plays the game better than they do. I really dislike the idea of changing the rules of the game because one billionaire is suddenly losing, when we didn't change the game when millions of middle-class families were losing.
And google has to obey to local laws and local copyrights and not just to US law.
The google book case is just another thing that's going to backfire at google. The ignorant attitude of google making foreign books (books written by authors where it is clearly illegal by the law do scan in the books and make those freely searchable on the net) available to all US citizens, even though the authors don't want their books to be available trough google, will only create more hate against google.
I personally know an author who is simply delighted that google is making his (out of print) book available. I think he will even get payment if someone wants the whole book.
You can't also steal something in a store, just because the store didn't tell you not to steal something!
This seems very much the same as walking past a newsstand, glancing at the headlines of assorted newspapers and magazines, perhaps flipping through one or two, to see what's happening. The stand owner is "giving away" the headlines, but how else are people going to know about the publications?
By similar logic, a newsstand owner should have to pay just to place a newspaper on display. Where has Murdoch been all those years while this robbery was occurring?
But now I don't want to criticize Google, nor do I want to be Murdoch's advocate. I just notice that the newspaper critics don't look at the issue from publisher's perpective, thus making their articles a bit one-sided.
Then aren't we done. Fair use and all that. It doesn't matter what value Murdoch places to them. In fact, why should it matter? Should every teaser be worth $1,000,000? If you let the marketplace decide then what happens? Probably what we have right now, because at least one news producer will think that Google is providing a great service.
"but those 'headlines and teasers' do carry news value"
Newspaper vending boxes have a nice glass window in front so you can read the headlines and choose whether or not to buy the paper. How is what Google does any different at all?
What I don't get is why the newspapers don't shut Google down in their robots.txt and then offer up a "firehose"/API a la Twitter that anyone can get access to for a fee. Yelp, Delicious, Twitter, Flickr, these services all have APIs that can be used to incorporate their content into other services, on their terms. I know the guardian and the new york times are working on APIs, but they should offer up a full-text firehose version that a company like Google could pay a fair price for.
That's because their real motives have nothing to do with what they are saying.
You can't assume that they want what they say they want because, of course, they are completely capable of making it happens -- as you say, with robots.txt. It's trivial and I can guarantee you that they're aware of that fact.
What they are probably angling for is some kind of legal settlement, either a lump sum or royalties for when their search results are shown, or Google uses their content to compute rank, or some kind of special treatment, a special relationship with Google.
To really remove themselves from Google would be shooting themselves in the foot, and doubtless they know it.
Or, like you said, they want Google to pay for a firehose. But why would Google pay for a firehose when the content is right there on their sites, which Google indexes anyway?
People, you've got to be more multi-layered in your thinking. I know nerds are known to be more literal than most, but you can't go through the world thinking that other people are like you. You have to compare people's actions, with what they say, and figure out what the gestalt means. And you have to question a person's motives for saying a thing.
Speaking, writing and publicizing are not just low-bandwidth means of transferring pure information.
No human communication ever is.
Look, a web site is a web site.
Facebook doesn't let Google in to index all of it's data, this created an opportunity to eventually sell access to index it.
Newspapers could do the same, stop them from indexing their sites and sell them access to this firehose api, just like _every_ other data-driven startup.
No one flinches when Yelp offers and API with specific usage conditions. Newspapers should do the exact same thing.
They may be hitting an easy target but that doesn't mean they're being insincere in their comments.
Also, a lot of people in the "mainstream" may still not fully understand the situation, so sometimes the obvious still needs to be pointed-out.