Hacker News Reader

I just tried this, doesn't work on WSJ. Tried the user agents listed here: https://support.google.com/webmasters/answer/1061943?hl=en

View on HN

Isn't that what the entire article is about? That WSJ no longer gives access to Google users (and bots)?

No. It allows Google bots to see full articles, but shows only the first paragraph or so to non-subscribers. Even if they're coming from Google search results.

However, I don't see cache links on Google :(

Edit: Oops, I'm wrong. The article does say that the Google bot only sees the first paragraph or so.

No. You're wrong, as it states in the article:

"The reason: Google search results are based on an algorithm that scans the internet for free content. After the Journal’s free articles went behind a paywall, Google’s bot only saw the first few paragraphs and started ranking them lower, limiting the Journal’s viewership."

Yes, you're right. I got confused by all the discussion about Google checking for cloaking by comparing results using different user agents.

So maybe this is why there's no Google cache.

Also, if Google can only index the first few paragraphs, the results are much less comprehensive.

Reply on news.ycombinator.com