Belgian newspapers: Google blocking us on searches | Hacker News Reader

Belgian newspapers: Google blocking us on searches

google.com

39 pointsrobtoo50 comments

The newspapers went to court asking for broad relief (unlisting from google results), instead of putting in place proper META tags and /robots.txt to direct what may be, and what may not be indexed and cached. The court handed them just as much of the proverbial rope as they asked for.

Google's web crawlers asked the WWW servers to GET, were served with HTTP 200 OK, found no prohibition in /robots.txt and so followed the standard protocol.

Note Google Web Search and Google News use separate bots (Googlebot and Googlebot-News) to give website owner full control.

I can't decide whether I find things like this hilarious or disturbing. They could have fixed it with engineering, but they were either so uninformed or incompetent that they tried to fix it with a lawsuit.

Or maybe one incompetent overrode them all.

Manager: Evans, get in here! Why are our news articles all over The Google?

Evans: Um, sir, they index our -

Manager: I don't want to hear your technical mumbo jumbo! Get legal on this to get it fixed YESTERDAY!

Evans: There's no need, it's literally a 5-minute fix, I could just add an entry to robots.txt while the coffee is brewing...

Manager: What did I just say about mumbo jumbo? Get out of here, and send in Stevens from Legal on your way.

Again, the lawsuit was over Google News' display of lead paragraphs and other excerpts from articles. It didn't pertain to the general search engine.

The lawsuit may not have, but the ruling certainly and explicitly does.

That's incorrect. The court's ruling specifically pertains to the use of article content in the Google News aggregator and its storage in Google's cache. There is no mention of simply linking to articles in the search engine, nor would there be reason for one since the lawsuit was over the use of copyrighted article excerpts, not hyperlinks.

What Google is doing is akin to the punishments Microsoft used to dole out in the 90s, such as increasing Windows OEM licensing fees or threatening to revoke licenses entirely. Google is playing a dangerous game in monopoly-conscious Europe.

Google uses separate crawlers for its search engine and its news service. Blocking the news service really would have been a trivial edit to their robots.txt file.

The lawsuit was over the use of lead paragraphs in the Google News aggregator, not the general search engine. Google is intentionally misinterpreting the verdict of the lawsuit in order to punish those who might challenge the company. Remember that Google also defied the court order to publish the result of the lawsuit on its Belgian sites, so it's obviously not afraid to thumb its nose at the courts.

"Nice online newspaper you've got there. Hate to see something happen to it."

> Google is intentionally misinterpreting the verdict of the lawsuit in order to punish those who might challenge the company.

Even if we stipulate this, how is that bad? Should Google be forced to deal with companies that screw with it? How far do you want to take that idea?

You're asking me how it's bad that Google is using a strong-arm tactic to punish content-owners who sued it? Surely, I don't need to answer that question, especially with Google's monopoly troubles in Europe. Google should let things be, respect the decision, and move on.

I'd also like to mention again how incredibly lame it is that my comments are being voted down in this discussion. All I did was correct people who assumed the lawsuit was pertaining to ALL search engine links. I'm at a loss as to what I wrote that warrants negative votes.

Didn't you notice every second link here is about Google? It's not permitted to say anything bad about Google here.

> You're asking me how it's bad that Google is using a strong-arm tactic to punish content-owners who sued it?

Yes, I am. If you sue someone, can you really expect them to be nice to you afterwords?

The newspapers went to court asking for broad relief (unlisting from google results)

They went to court over Google News copying excerpts from their articles, such as the lead paragraphs, and displaying them in the Google News aggregator. The lawsuit didn't pertain to the general search engine.

That might have been the intent of the newspapers, but that wasn't the court order their lawyers got them.

That's incorrect. You can read the ruling for yourself:

http://www.copiepresse.be/pdf/Copiepresse%20-%20ruling%20app...

The decision was pertaining to the use of lead paragraphs and other excerpts from copyrighted articles in the Google News aggregator. It didn't refer to plain links in the general search engine.

I see that my comments in this discussion are being voted down for pointing this out, which is bizarre. I'm simply stating the truth and correcting posts that mistakenly assume the lawsuit was in reference to ALL links in the search engine. Voting me down is incredibly lame.

I believe that Google is deliberately misinterpreting the decision, likely to remind content-owners that Google controls their source of online revenue and to therefore back off. Google has a history of being defiant with regards to this lawsuit, ignoring a court order to publish the court's decision on their Belgian websites, so they would surely have no problem doing something retaliatory under the guise of following the court's decision.

OK, so I read the PDF you linked to. Page 48, under the heading "V. ENACTING TERMS", point 3:

--- Begin Quote --- Reforms the challenged judgment to the extent that it ordered "Google to withdraw from all its sites (...) all the articles, photographs and graphic representations (...)";

Ruling again on this point only, reforms the injunction as follows: Orders Google to remove from the Google.be and Google.com sites, more specifically from the "cached" links on "Google Web" and from the "Google News" service

--- End Quote --- (There may be some errors in the copy above and please excuse my pathetic formatting).

See the thing is you are arguing about what the lawsuit was about, which is different to what the judgment was.

It says there in black and white "to remove it from all sites".

Or am I just reading that wrong?

See the thing is you are arguing about what the lawsuit was about, which is different to what the judgment was.
It says there in black and white "to remove it from all sites".

Or am I just reading that wrong?

You're reading it wrong. It was a copyright infringement lawsuit over the scraped content displayed by Google News and in Google's cache, so the court's ruling was to remove those infringing materials from those services, not to remove the hyperlinks to the original articles in the general search engine.

> I see that my comments in this discussion are being voted down for pointing this out, which is bizarre.

Before posting the above link, You failed to provide quotation supporting your statements, and there were quotes saying otherwise :-)

Let's see, the original ruling went:

  Orders GOOGLE to withdraw from all its sites (notably Google news
  and "cache" google or any other name) all the articles (...)

(page 9, 2nd paragraph)

However, there is amended ruling listed on page 47 reads:

  Orders Google to remove from the Google.be and Google.com sites,
  more specifically from the "cached" links on "Google Web"
  and from the "Google News" service, all the articles (...)

which confirms your story. Thanks for being persistent!

----

At any rate, this lawsuit is bullshit, and should be called out till they blush. The webserver has been likened to house left with doors wide open and robbed in absence of the owner (page 36.). Seriously?

What The Copiepresse actually did was like having welcome guests for a long time, and then suddenly suing them for enjoying themselves.

The argument of META tags and robots.txt was raised:

  In 2006, when the action was filed, the publishers
  had not activated these tags, which allowed Google
  to inventory all their pages, notably "cached".

but rejected on weird reasoning:

  However the parties do not explain how it would be
  technically possible to, via these tags, prevent Google
  from reproducing ((the content)) in the "Google News" service.

(page 35.) I say, it's the same as with law or court orders: doesn't technically prevent, just informs what is allowed and what is not.

----

For a wider picture: the newspapers joined (by publishing public websites) a well established space: the Internet. It takes serious incompetence -- or malice -- to move somewhere and not follow well-documented, mutually-benefiting local rules.

Let's be clear: Google is no schoolyard bully here -- even if it was elsewhere the other day -- and newspapers are no victim. If anything, it's newspapers that advanced from large, slowly grinding company to a fast, focused start-up and found: their playbook hopelessly inadequate, their narrow field of view constraining.

You failed to provide quotation supporting your statements, and there were quotes saying otherwise :-)

The lawsuit is public knowledge, with plenty of articles written and a decision published online. I think we both know that I was getting voted down because I was correcting misinformation about a court decision against Google. :)

If you started a newspaper that did nothing but copied the headlines and lead paragraphs from, say, the New York Times, how long do you think it would be before you got a cease-and-desist letter in the mail? That's copyrighted material.

There's a rule stating that content-owners are supposed to let another company take their work and make money off of it? What rules are you referring to specifically?

Let's be clear: Google is no schoolyard bully here -- even if it was elsewhere the other day

That's arguable, but more importantly, it's a dangerous maneuver for Google in anti-monopoly Europe.

> I was getting voted down because I was correcting misinformation about a court decision against Google. :)

Can't speak for others, but I've voted you down exactly because you were making strong claims -- opposite to sources available to me -- without support (till you linked to that document). One of my mistakes, eh.

To the wit: I don't particularly like Google. But I strongly believe each case should be decided on its own merit, not on popularity (or otherwise) of defendant.

> If you started a newspaper that did nothing but copied the headlines and lead paragraphs from, say, the New York Times, how long do you think it would be before you got a cease-and-desist letter in the mail?

If I started a website that did way more than just copying titles and lead paragraphs from wide range of media sources, how long do you think I'd have to explain to various folks it actually is legal? It's the videocassette recorder lawsuit history all over again [1]. Significant non-infringing uses.

> There's a rule stating that content-owners are supposed to let another company take their work and make money off of it? What rules are you referring to specifically?

You have twisted it strangely @_@ Nobody asks the owners to let others use their content, only to indicate their preferrence.

There is a rule that what is served with HTTP 200 OK (and has such and such META tags and permissive /robots.txt) to a robot is allowed for that robot. Most websites need their content to be included in search results, news or not. Thus the reasonable default.

There is expectation of due dilligence. If you by mistake leave your home open once and somebody robs it, it's a robbery, plain and simple.

But! If you invite somebody to live with you for a good stretch of time, with mutual benefit, you can't then claim they profited against your will. It's your conscious decision, plain and simple again.

The newspapers did the later.

Web crawlers are fact of life, pre-dating the newspapers' webservers. There's an agreed-upon protocol for dealing with them. You either show due dilligence (which, nb., is easy and cheap -- the document you linked states clearly the newspapers put proper META tags in place at some later point) or you admit mistake and clean up your mess yourself.

Want a car analogy? Nobody asks drivers to turn only left. But they must indicate turns unambiguously with signals. The newspapers lit the wrong signal and now want their scratched car reimbursed.

----

At any rate, thanks for bringing the actual ruling to attention :-)

----

[1] http://en.wikipedia.org/wiki/Videocassette_recorder#Legal_ch...

Looking at the text of the ruling, I can see how someone might interpret it either way, either as having to eliminate the links they have "cached" or the links to a "cached" version of the website. I think that if I was a legal counsel to Google I would probably advise them not to do any linking at all in the face of that ambiguity and the large penalties for violating the court order.

Equally important, I'm having a hard time imagining how it would be reasonably possible for Google to only link to the articles. A hyperlink is a combination of both text and the link, and the text has to come from somewhere. Usually it is the title of the web page that is linked to, but then Google would still be stealing content from the newspapers. I suppose they could just use the URL itself as the text of the hyperlink, but then we get to the second part of this issue. Complying with the order and still providing links would require rewriting a lot of Google's front end code just for this one specific case. Given that Google already has a process to block sites entirely at the back end it would be a lot easier to just comply with the court order that way, and probably the only way that they could do it quickly enough.

OK, it says they filed a lawsuit against Google because they didn't want Google to LINK to their articles without paying a fee for linking. They won the lawsuit in court and so Google removed links to their articles from their search index, as they requested, and now they are mad about it. Why is the story "Google blocking" them. Google's only doing what a court ordered after the papers sued asking for that.

As I understand it, they wanted Google News to stop linking to their articles; not the regular search results.

From court order, via http://www.chillingeffects.org/notice.cgi?sID=2160

> Order the defendant to withdraw the articles (...) from all their sites (Google News and "cache" Google or any other name (...)

The other articles linked from http://slashdot.org/story/06/09/18/115203/Google-News-Remove... are down now, but I believe the original lawsuit was in that very spirit: remove their articles from all Google's websites. Not just from Google News.

Logical argumentation aside, Google's in no position to second-guess a court order ;-)

Why did Associated Press put such a negative spin to the story? We can only guess...

They sued because Google News was using lead paragraphs from their articles. Google is most definitely block them, because the court's decision didn't pertain to the general search engine. In effect, Google is punishing these content-owners for challenging Google's use of their content in its service.

The only comment that comes to mind: Don't bite the hand that feeds you.

The newspapers really didn't think that one through, did they...

I suppose to extend that analogy: the newspapers didn't believe they were biting the hand that was feeding. They saw it more as cutting off the other hand that pushed their heads towards the food.

Google is saying both hands are equivalent, but the newspapers are saying they want one hand to feed them, and the other to keep away.

I can buy that, actually. Sure, I don't agree with the initial ruling, but IMHO de-indexing the newspapers from the search engine was unnecessarily retaliatory. I can understand why Google did it, but having made their point, I think they should gracefully back down claiming ignorance. This is a clear misuse of their 'monopoly' on web search for benefit in other areas.

EDIT: Oh good, my Google fanboi-ness is not in vain. The court ruling actually ordered that links be removed from "Google Web [sic]" as well. Oh well, the joke's on the newspapers then, as it appears they asked for it! (see roel_v's comment: http://news.ycombinator.com/item?id=2776384)

Unnecessarily retalitory? I wouldn't be so sure. Don't forget that Google search results include selected content from indexed websites. The index itself actually contains all content, which Google may have believed to be a litigation risk given the vagueness of the removal request.

Google supplies the tools for the newspaper to have marked explicitly, using technology, the pages they want included/excluded from search, with no ambiguity. Instead, they chose to use vague, court language, with threat of a fine per link, to force google's hand. Personally, I don't know why google -wouldn't- drop them, permanently.

There is no monopoly on web search. Search works for everyone, specifically because there are established protocols and conventions that allow all these systems to work together. If your company chooses to ignore these conventions, I don't think it is unreasonable that you get ignored.

You don't have a -right- to be included in a web search. The reason google includes you (or any search engine) is because you are making a trade, they get better results which they can monetize via ads, you get more traffic, which you monetize through ads. If you opt out, why shouldn't they? What's in it for them? They aren't running a charity, they are running a business.

Your post demonstrates a misunderstanding over what the lawsuit was about. Publishers weren't suing to make Google stop linking to them. Google News was displaying full excerpts from copyright articles, sometimes entire paragraphs, and on top of that making money off that content through the display of ads. It was a copyright infringement lawsuit.

There is no monopoly on web search.

Google has over 65% market share in search. Second is Yahoo with only 16%.

> they want one hand to feed them, and the other to keep away.

They dont seem to be in a negotiating position to express "wishes" which then somehow have to be granted in the exact way they want them.

From http://www.copiepresse.be/pdf/Copiepresse%20-%20ruling%20app... Section V:

"Orders Google to remove from Google.be and Google.com sites, more specifically from the "cached" links on "Google Web" and from the "Google News" service, all articles..."

Depending on your interpretation, they were supposed to remove the newspapers from Google News and remove the "cached" links from Google Search results.

There's a € 25.000 euro fine on every violation of the court order. Google doesn't want to take a chance to leave room for even the smallest amount of ambiguity, so that's why they've removed all links to those newspapers.

The lawsuit was pertaining to Google News, not the general search engine. Google removed the links as punishment for challenging them in court, to show the content-owners that Google controls their revenue streams.

* EDIT: Oh good, my Google fanboi-ness is not in vain. The court ruling actually ordered that links be removed from "Google Web [sic]" as well. Oh well, the joke's on the newspapers then, as it appears they asked for it! (see roel_v's comment: http://news.ycombinator.com/item?id=2776384) *

That's in reference to the Google cache of lead paragraphs and other article excerpts. The lawsuit was not about general links in the search engine. Publishers sued because Google News was displaying lead paragraphs and other copyrighted writing in the aggregator.

If you started your own newspaper that copied all the headlines and lead paragraphs from the New York Times, it wouldn't be long before the New York Times sent you a legal letter.

Anyone have a copy of the ruling? Might just be the judge(s) misinterpreting what the newspapers were asking for?

Ruling (in French): http://www.ie-forum.nl/backoffice/uploads/file/IEF%20Hof%20v...

Can't copy and paste from there; relevant passage:

"...orders Google to removing the sites google.be and google.com, more precisely the links (in cache) visible through Google Web and the service Google News (...) of Belgian Francophone and German-language publishers represented by Copiepresse."

So yes, Google is interpreting the ruling correctly, there is no way to interpret it in such a way that the regular google.com shouldn't be blocked. I don't know if the newspapers asked for too much or if the judge didn't understand what he was doing; I didn't read the whole ruling.

> there is no way to interpret it in such a way that the regular google.com shouldn't be blocked. I don't know if the newspapers asked for too much

I think theoretically it would be possible if and only if they specialized Google's crawler and rendering to only render a link (no preview, no cache link) for these sites and these sites only.

Why they'd bother doing that is anyone's guess.

The lawsuit was over the use of lead paragraphs and other article excerpts, specifically in the Google News aggregator, not plain links in the general search engine.

Yes of course, but the ruling was broader than that, that is the whole point of contention.

Update on this story: Google has said they will start indexing the pages for the regular search results again asap.

Ruling in English: http://www.copiepresse.be/pdf/Copiepresse%20-%20ruling%20app...

In the 2006 injuction, Copiepresse had originally asked the judge to "order [Google] to withdraw the articles, photographs and graphic representations of the Belgian publishers of the French and German-speaking daily press, [it] represents, from all their sites (Google News and "cached" Google or any other name) from the date at which the ruling is served, under penalty of a fine for non-performance of two million euro per day of delay".

And the relevant part of the ruling was thus:

"Orders GOOGLE to withdraw from all its sites (and more specifically from Google News and as far as the Google web search engine is concerned the visible cached links) all the articles, photographs and graphics representations from the authors in respect of whom the intervening plaintiffs can prove that they retain the rights".

In another part of the document, it states that "Google News does not confine itself to placing hyperlinks but reproduces significant sections of the publishers' articles".

If you read the entire ruling, it is quite clear (as far as such a legal document can be clear) that only the copied content on Google News and the cached copies on the search result pages were problematic. Copiepresse or the judge had no issue with hyperlinks.

By taking it further, Google is playing a dangerous power game. It seems to me that this will inevitably lead to a EU anti-trust case.

Maybe I just don't get it, but why do they want to be included in web search but not in Google News? Their actions seem bizarre.

Both are noindex-able separately, correct?

tl;dr: Yes they are indexed separately, and yes it's up to site owner to decide which content may be indexed for Google News -- via proper META tags and robots.txt.

There is an important difference in UI between Google Web Search and Google News: the Web Search presents a very short snippet of text and link, while Google News presents a bit more of the news content. Effectively, Web Search directs viewers straight to the original website, while News lets them just browse Google News, sending less traffic to the original website. [1]

Which means Google News may provide less ad revenue to original website -- probably the crux of the problem here.

To handle that, Google provides two different bots: one with user agent `Googlebot', the other with `Googlebot-News'. If a website wants its content only in Web Search, and not in News, it's supposed to return proper /robots.txt -- and that does the trick. Way quicker and cheaper than any lawsuit!

http://googlewebmastercentral.blogspot.com/2009/12/new-user-...

----

[1] but hey, less is more! whoever makes it to the original site via Google News has clearly show elevated interest in the news piece, and thus should be worth more to ad providers.

Short answer, one looks like a newspaper.

The longer answer is that the newspapers are struggling financially, Google is not struggling, when snippets of content appear on the Google news page the newspapers believe they are 'improving' the Google brand without being compensated. They sued to have either Google pay them, or remove them from the News site.

What they fail to realize (and Google no doubt knows) is that if you aren't listed in Google you're not going to get nearly as much traffic (hence 'profit'). So Google apparently took a literal interpretation of the order and struck them from the books as it were, complete removal from the index.

So the newspapers have to make the following evaluation, 'what is it worth to us to be in Google's search results?' And is that value more or less than what we think they owe us for having our results on their news page?

Google clearly understands that its value for them to be in the search results and I believe they would say off the record that the value of being in the results is worth more than the ad revenue they get from the occasional story that makes it to the front page of Google news.

Sort of extortion, sort of natural consequences. Humorous though.

The papers are back according to this news article:

http://babelfish.yahoo.com/translate_url?doit=done&tt=ur...

They also hint that the complaints have been lifted by the papers.

Yes, they are back in the index: http://www.pcmag.com/article2/0,2817,2388635,00.asp

And in a simple test, it looks to work too: http://www.google.com/search?q=copiepress+lalibre+critique The second link (lesoir.be) is one of the newspapers.

This is not a case of technical ignorance, though it's hilarious to pretend it is. The French-language newspaper association wants to receive money from Google (for the ads displayed next to their content) and figured it would be easier to pressure them through the courts than through other means.

The lawsuit was over the use of copyrighted excerpts from articles in the Google News aggregator and cache, such as lead paragraphs.

Google is trying to pretend that the court ruled all hyperlinks to be illegal. Google was defiant before over this ruling when it ignored an order to publish the results of the verdict on its website, so it's not far-fetched to believe Google is once again throwing its weight around and reminding people of the power it wields over the livelihoods of content creators.

   User-agent: Googlebot
   Disallow:

   User-agent: Googlebot-News
   Disallow: /

Reply on news.ycombinator.com