Hacker News Reader

Comment by leorocky | Hacker News Reader

This absurdity is found all over the internet and it's the result of Google's crawling policy. You can do the same kind of thing with nytimes.com articles and other websites that want their context indexed by Google but at the same time restrict access to it. These absurdities are made and work because most people do not understand how to use the software they have.

View on HN

Yup. They need to show all the information to the Google bot in order to rank higher on the SERP. If they did not show the full name to the bot, this would not be possible. But they need to have the same behavior to any one visiting their site from Google search results.

I just use a incognito browser when I need to lookup someone on LinkedIn unless I want them to know that I visited their profile.

I think Quora changed their behavior a little since the earlier days? They now always have the first answer visible to everyone.

I thought Google heavily penalised sites that made content available to the Google bot but firewalled content for everybody else?

I think there is a penalty for showing Googlebot something other than what people can see, but these sites all use tricks to get around it by allowing users to see the same thing as Googlebot in various circumstances. For example Quora has the ?share=1 URL variable. The Wall Street Journal allows users to read a limited number of articles for free if they clicked in from Google (tracked via a cookie). The New York Times allows free reads from anywhere for a limit, but also tracked via a cookie. LinkedIn blocks their registered users, which is probably the worse system ever.

This is subtly different for Linkedin. They DO show the same thing to Googlebot as logged out users. Registered users end up seeing a different page which Google bot has no way of accessing because it does not have an account.

So GoogleBot can't scrape more than 10 NYT articles per month, right?

Googlebot doesn't maintain a nytimes.com session, so it can crawl an unlimited number of articles same as you could if you tell your browser to not accept cookies from nytimes.com

Reply on news.ycombinator.com