Hacker News Reader

Comment by fngjdflmdflg | Hacker News Reader

I'm surprised it took them this long. I like many others used it to view paid articles for free. I imagine paywalled sites didn't like that and told them to shut it down.

View on HN

Wasn't that NOARCHIVE?

https://webmasters.stackexchange.com/questions/679/how-do-i-...

I'm not so familiar with this area but my guess is that if you turned used noarchive, Google would not cache the page at all and therefore would not be able to use the text in your page as keywords for search results. So most sites therefore did not use noarchive because it improved discoverability/SEO to allow Google to cache your site. This is just a guess though and what I always assumed to be the case. This seems to be the case though because the cached versions would often contain the entire article for free, which makes no sense unless they were doing it for SEO. For example you could use this trick to read any nikkei article.

Nah, it's not a Google thing although Google honored it. Here's a reference to the Internet Archive using it:

https://archive.org/post/31561/robots-archive-noarchive-meta...

I didn't mean to say that noarchive is only a google thing. My only point was that my assumption is that until now if Google didn't cache your site its contents would not be used for SEO.

I didn't even realize this existed. Now it's too late to enjoy it.

You can still set your user-agent to Googlebot.

They check for client IP. True Googlebot always comes from 66.249.*.*

Yeah I was like "surely it can't be that easy." So I went to try, and no, surely it is not.

Sometimes you can get lucky though – I know at least one forum that requires registration even just for viewing posts, but lets Googlebot through and only checks the user agent.

https://archive.is often works for paywalled sites.

Reply on news.ycombinator.com