Wait, Google Sent Me | Hacker News Reader

Wait, Google Sent Me

chrome.google.com

502 pointsjashkenas237 comments

Open articleView on HN

I have actually made a bookmarklet that allows you to do the same without installing an extension.

https://umur.io/paywall-bypass-bookmarklet-come-from-google/

Related HN discussion: https://news.ycombinator.com/item?id=9134118

That is pretty slick. I prefer something lightweight like this.

Very nice approach. I didn't know that Google had that auto-redirect feature.

But it's not working reliably for me. Frequently, when I click the bookmarklet, it gets stuck on a blank page. Does it work every time for you?

> I didn't know that Google had that auto-redirect feature.

It uses the functionality of the "I feel lucky" button. It will only work if the site has been indexed by Google, but that's usually the case with news pages that show such fake paywall messages.

Btw. please Google fix your frontpage, you serve an unformatted text message (IE9-11): http://s7.postimg.org/52fu9wbdn/google_bug.png

Is it Google or IE that's broken? Other browsers seem to be coping with it - I know these big sites tend to serve really mangled markup/code however.

As you can see it's still loading. Let it load.

Thanks.

As @frik noted, it uses "I'm feeling lucky" feature of Google. It requires the page to be indexed to work. If the page isn't indexed then it won't work. But since the pages you'd like to see are indexed almost always, it shouldn't be a problem.

No, that's not what I mean. I mean that the bookmarklet is working intermittently for me on any page. I'll click it, and sometimes get redirected through, and sometimes end up on a blank white page. That doesn't happen to you?

I see a blank page for a second than I get redirected to the actual page. Could you also hit me on twitter @umurkontaci to help me diagnose?

FYI: The test link[1] doesn't work for me. It either reloads a white page, or reloads the page but it is still firewalled.

[1] https://go.umur.io/wsj/

That's interesting, I see a white page for a second or two and I get redirected. If something else is happening for you, would you mind sending me a DM on twitter @umurkontaci? I might be able to fix it.

Haha very nice solution

I am going to use your bookmark. Thank you for making it.

It would be nice if Google could make some of the Chrome permissions a bit more fine-grained.

This one needs to "read and change your data on all websites", but with a better extensions architecture could be something much less scary like "read and change your data on the current website whenever you click the extension's icon".

Yeah, especially that last part. It would be great if all permissions started with "Upon page load..." or "Upon clicking the extension icon..."

That granularity exists -- it's called activeTab: https://developer.chrome.com/extensions/activeTab

(Though it might not be possible to modify the referer with activeTab alone :/)

Nifty tip. Let's try it, and see if it works.

Edit: Sorry bud. I gave it a shot, and `activeTab` doesn't allow you to modify the "Referer" header.

Yeah, if you really want to do this, you can use chrome.permissions with "optional_permissions" set in your manifest requesting all urls -- then call permissions.request() on invocation to add a specific host.

See: https://developer.chrome.com/extensions/permissions

You could dynamically (and temporarily - use chrome.permissions.remove when the tab navigates or closes) request the webRequest permission for the current host when the user clicks the browser action icon. Yes, that would mean clicking again to confirm whenever the user wants to use the extension, although it's possible that Chrome will remember what permissions have been granted and not prompt the user for future use on the same site.

There is always the option of not messing with the request at all. Instead actually have it redirect through google. For example you could have the background script do:

  chrome.browserAction.onClicked.addListener(function(tab) {
    chrome.tabs.executeScript({
      code: 'window.stop(); window.location = "http://www.google.com/badurl/' + encodeURIComponent(tab.url) + '";' 
    });
  });

and then you can have a content script run on "http://www.google.com/badurl/*" which just does:

  window.location = decodeURIComponent(window.location.pathname.split("/")[2]);

Now that is super brittle compared to your solution, but it does work(for the sites I tried, some may require you to actually fake a search query). It only requires the activeTab permission and content script permissions on one page.

Alternatively, it would be good if extensions could have a whitelist and/or blacklist of sites that they are enabled on, with most extensions having a default blacklist of all financial sites, banks, brokerages, email accounts, etc. Perhaps also create a way for corporations to blacklist their intranets. It makes me very nervous that if some extension owner sells out to the wrong person, suddenly they could see my private information.

Agreed -- see this never-ending Chrome bug: https://code.google.com/p/chromium/issues/detail?id=177351

Yeah and then you could extort people for money like the AD-Block guy!

I haven't used AdBlock for a year or so now, but I'm fairly certain sure his model was donationware. Did something change recently?

The AdBlock Plus guy gets paid by companies to put their site on the "unobtrusive ads" list -- those ads don't get blocked for the majority of AdBlock Plus users.

The issue is that if it can modify one page, it can use that to inject code to request data from any web site. Those requests will contain your cookies for those sites so your data can be revealed.

I'm not expert on security, but wouldn't that only apply to sites that have XSS vulnerabilities? The extension script only needs to have the same trust level as the top-level HTML in order to do its thing, in most cases.

At the moment, the ask is that users put 100% trust into some extension from the Chrome Web Store. It's not clear that Google does much to ensure that the companies are even who they say they are. It's a completely unreasonable ask, in my opinion. Then, in other areas, Google is hyper-sensitive about security. Eg. Chrome, under some circumstances, won't even let users download zip files anymore without trying to intervene. (https://code.google.com/p/chromium/issues/detail?id=423217)

Attention, everybody who publishes on the web:

Selectively requiring authentication (based on referrer, user-agent, cookies, etc.) is not a supported feature of the internet. If you want to go ahead and implement it anyway, that's fine - but it's pointless to get mad when it doesn't really work the way you hoped it would.

More generally, everyone who publishes on the web should understand that user agent is to be taken literally and at face value. The browser does what the user tells it to do, which may not be what your server asks it to do. A good browser will do things like lie to your server on behalf of the user when you try to get it to act counter to the user's interests.

Sure it is. Depends on your definition of authentication. But it's totally a "supported feature of the internet".

This plugin isn't fundamentally different than spoofing a session cookie.

Source code: https://gist.github.com/jashkenas/9a4d9e648eb11aab394e

Could you give more information on how do some websites react to a google referer?

The design intent is almost certainly using Google's "first click free" policy, which primarily applies to newspapers like the NYT or WSJ. Google back in the day struck a deal with the print media: they would remain in Google's index if and only if clicking from a Google search to their articles circumvented their paywall on the first click. If they put only partial content or a full paywall, they'd be delisted. It was not exactly a secret that this would mean that Google would rank scrapers higher for NYT articles than they'd rank the NYT.

Thus, when someone posts a paywalled link to HN, someone inevitably posts the advice "Google the title and click through." (Other than, say, "If you're a working professional, just buy a WSJ and NYT subscription.")

So it's not so much that they intended to have a "first click free" policy in the first place but that they are forced to do an exception in order to remain in Google's index. That's interesting, I always wondered why didn't they apply the same policy to being linked in Reddit or HN.

Actually, some do. NYT for example allowed (maybe they still do) incoming links from Twitter & FB to bypass paywall

I am a working professional but I have always resisted buying a subscription because I don't want one single publication to become my de facto news source.

I would love if I could pay $20 to $30 per month to some aggregator that lets me read articles from Al Jazeera, NY Times, WSJ, BBC, and a selection of other diverse sources.

I like Circa News (take my money please!) because they aggregate news from a variety of sources and present news items in small bit sized chunks listing facts or statements. They also link to the appropriate sources. edit: and you can follow news stories to get updates when new information is available.

Never heard of Circa News before, thanks for mentioning it.

https://circanews.com/ for the interested. I use the app exclusively though and haven't spent much time on the website.

They're either shutting down or being acquired soon: http://fortune.com/2015/04/30/news-app-circa-buyer/

That's unfortunate. I'd pay for it even if it were free otherwise because I like the service so much.

That's a damn shame. Their (android) app has unfortunately become significantly less useful due to technical reasons lately. I always did wonder how they would monetize.

If you get a chance you should check out BriefMe -- pretty much that, except free, with a very clean UI. http://www.getbriefme.com

Oh my, that's a terrible site. So I have to watch a full page video to find out what I might want to install?

Weren't they the same publishers that were suing google for linking directly to their articles?

No. They weren't. Some mis-guided ones tried.

One issue with the subscriptions, at least with the NYT, is that their pricing is a little crazy. The phone version is $15, the tablet version is $20, and both together are $30. It seems... excessive. WSJ at least (used to?) offer a single subscription that gives you access on all devices.

Not to defend the NYT craziness, but that plan somewhat exists in the weekend print edition and should save you money, too. Subscribe, check the box in your account to have them donate your paper copy to a school or library, and enjoy access on all of your devices.

I didn't see that option when I signed up a few years ago. When I look at my account now, the only place I see that option is when requesting Vacation Suspension. Do you happen to know if there's a place on the website to set this indefinitely? I guess I can call.

I'm on permanent vacation as far as the NYT knows

Gotcha, and it sounds like that doesn't affect your digital access. Thanks for the tip. At least 50% of time, we just throw away our physical Sunday paper without ever looking at it. It's ridiculous, but I couldn't find any friends who wanted it.

Many news websites will pop up a paywall if you visit a link directly, but won't if you arrive from a Google search.

The reason for this being that Google results penalize sites that show content to the crawler and then hide it from people who visit the page (ie non-subscribers). If they want it in the index, it has to be actually viewable.

The alternative would be to not show paywalled content to Google's crawlers, and have them entirely left out of search results.

I've seen some websites highlight your search terms for you. It's kinda useful, but it's also kinda creepy at the same time.

Did someone make a Firefox version yet?

You can achieve the same thing with the RefControl add-on, which allows you to set a custom referrer for a particular site:

https://addons.mozilla.org/en-US/firefox/addon/refcontrol

Fantastic tip. For anyone interested in trying this out, use http://news.google.com as the "custom" referer.

Screenshots and help here: http://www.renjusblog.com/2010/01/hack-read-wall-street-onli...

Thanks, this worked perfectly

Perfect! Thank you!

For sites that has x free articles/month, I just discovered Private Tab[1] which makes it easy to turn an existing tab private or force a bookmark to always open in a private tab.

[1] https://addons.mozilla.org/en-us/firefox/addon/private-tab/

Self-Destructing cookies extension also keeps the read count at 0 after you close the tab. As an added bonus the extension should also help prevent you from being tracked across the web by various advertising/state agencies.

here's a port i cobbled together from some firefox addon demos in 15 minutes. tested to work on wsj

https://github.com/zethraeus/wgsm-ff/blob/master/gglsntm.xpi...

(consider reading the source instead of just trusting it)

I sent you a pull request improving your code – only making the change apply to the current tab, and changing the Icon Name ;)

Interesting. I posted a Chrome extension that did the same thing a while back (WSJ specific) and HN unanimously told me I was in the wrong, so I took it down.

https://news.ycombinator.com/item?id=8109987

This time the comments seem to be mostly positive.

Apparently having a few javascript frameworks under your belt can put you on the right side of the law.

Now there's a stinging criticism of HN.

Sorry habosa. You're not popular enough to do disruptive things.

Too true. I'd love to see what Hacker News comments would look like if all comments were anonymous for the first 6 hours...

Hm. That is an unbelievably charming idea for an experiment. Does HN ever do April Fools' day experiments? That would seem worth it for the introspective value alone...

I think you should suggest this idea to the admins. I know PG stepped down but this seems like something he might have been in favor of.

I really like this idea.

I'll chime in with that it's an interesting idea, OTOH my gut feeling is that most people won't come back to read article comments after this initial 6 hours or so — hence you might as well essentially make the user handles anonymous altogether.

Just like that movie 'The Purge'.

Wow, those comments are really odd. Every time a WSJ article is submitted here a comment contains a link to avoid the paywall, and every single time people are grateful.

I can't help but wonder if that was a bit of brigading...

Every time there is also somebody complaining that we shouldn't evade pay walls at all.

But that's indeed a really odd set of comments. While there is some criticism on this thread here, it's way more down to Earth, nobody is talking about theft or anything.

Maybe people feel that people should only avoid paywalls in a random and haphazard way, and not a systematic way, or that people should avoid paywalls whenever they want but not make it convenient for the general public to do that?

Is there a name for ethical notions related to whether people make a habit of something, or whether people help make something convenient?

It is called "abusing a privilege" , and most elementary school children are taught not to take all the cookies from the jar just because they are invited to take one.

It was your fault for listening to the comments. A golden rule is to never listen to the haters. Do what you think is best.

In that post you didn't advertise that the "pay"wall can be circumvented just by Googling. It's possible HN perceives an ethical difference between that and a paywall where you actually have to pay.

Agreed. I think the commenters in the other thread might not have understood how the extension worked. While the end result is the same, it feels like there's a difference between "circumvent a paywall" and "pretend I'm coming from Google search result".

1) Many people are ethically opposed to circumventing paywalls.

2) Many people are ethically opposed to paywalls treating certain traffic as special or different.

As Jeremy says below: "What's not so great is creating an internet where if you use our Trusted Corporate Partner™ you're allowed to view a public webpage, and if you come from anywhere less privileged on the internet, you're not. That's not sustainable. Lest we fall into a vortex where you're only allowed to browse the public web by logging in through Facebook. I wouldn't want to be online in that world."

I agree. I think the difference in responses from HN commenters is about which ethical evaluation people are focusing on, #1 or #2 above.

If it makes you feel better, I feel that both extensions are wrong. If you enjoy reading the WSJ, you should pay for it. They do some really great business reporting and it's not that expensive.

> I feel that both extensions are wrong.

I feel that discriminating based on referrer is wrong. Or IP address. Or geographic location. Or user agent.

More than that, discriminating based on those things is really flimsy.

A website should provide the same content regardless of what browser you're using or where you're coming from.

I'm not saying I think the WSJ's paywall is a good idea (I don't), but if they want to try to make money to support their (largely very good) journalism that way, I don't think it's very nice for someone else to publish a tool that automatically subverts the paywall for countless other users.

This tool saves the 3 seconds it takes to highlight the article title, 'Search Google for ...', and click the link in the search results.

I don't follow. Would using the plugin be less acceptable if the paywall were harder to get around?

Yes, definitely.

> Would using the plugin be less acceptable if the paywall were harder to get around?

I don't see the plugin as getting around their paywall. I see it as getting around moronic web development.

For example, if a site has unbelievably intrusive advertising and I install an ad blocker. Or a site pops up a register for our newsletter modal and it's blocking the content and there's no way to close it ... so I open up the console and remove the dom node entirely so I can see the article.

In other words, stupid shit that 99.9% of people don't know how to get around, but a web developer can get around in a few seconds.

That's not hacking or bypassing anything, that's just moronic web development.

If you want to put up a paywall, that's fine. A wall is a wall. If you make a hole in the wall on purpose and people walk through that hole, don't blame the people ... blame the wall.

Or how much you're paying?

Fine you can have a paywall but then you won't get the free organic traffic from Google. WSJ tries to have it both ways.

I guess I just disagree. Right now if WSJ comes up in your google results, the link works and you can read the article. If link didn't work and paywall appeared instead of the article, that would be a problem (which is why Google outlawed it).

Would you prefer it instead if WSJ.com just removed themselves from Google? And applied the paywall to everyone? I don't see how that's better. Or is the complaint that any paywall exists anywhere?

If you discover content via Google, you can see it. If you discover content via HN, you can't. Anyone who believes this is ok is pretty much saying they are opposed to net neutrality.

This has nothing to do with net neutrality. While it may be an important issue, conflating it with net neutrality is unhelpful.

Traffic flowing through one discovery channel is free, but traffic flowing through another isn't. While technically separate from net neutrality as it's happening in a different place in the networking stack, it's the same principle: if it flows over the network, it should be equal.

> Traffic flowing through one discovery channel is free, but traffic flowing through another isn't.

That really misrepresents the case. Its not a difference in traffic costs, its a difference in whether the edge provider is willing to send its content over the network to you. And they are willing to do so if you are paying them directly, or you are sent by someone who is paying them with a certain quantity of exposure.

While you could probably formulate a single principle that would support both what you are calling for here and net neutrality, it probably would be a controversial principle even among people who support net neutrality, and wouldn't represent what many people who do support net neutrality support. Claiming that if someone doesn't agree with you on this they must oppose net neutrality is simply wrong, and unhelpful to either your cause of net neutrality.

That's not the same thing:

Net neutrality is about the service provider not getting involved in what the service carries.

A content owner can freely decide who can/cant view their content, that's completely up to them. Same way we have regional rights for video content.

Just because we want it to be equal doesn't take their rights away, regardless of consumer perception, etc.

"We don't serve Jews here."

What are both of the "ways"?

The content is the same, it's the business model of the publisher that's different: ads vs subscription.

Google is just serving as an index, you can't read it on Google itself so why should they be able to discriminate content providers based on business model?

This is a strange comment. Google indexes and searches lots of things you eventually have to pay for. Google's a search product. Seems to me the perfectly natural state is "free organic traffic" from Google and the WSJ getting paid.

> it's not that expensive

It's actually free. At least the first paragraph or so is. And a lot of the personal finance articles are totally free. About 99% of the time that's good enough for me.

So I don't bypass the paywall. I go to WSJ on a daily basis and simply read the free intro text.

I enjoy the WSJ enough that I would gladly pay them, except they play the usual pricing games, which I detest. I wish they just gave me their "best" price, and never tried to jack up that price to whatever they thought they could get away with at renewal. On a few occasions I've had a 1 year subscription for IIRC perhaps $99 but at renewal they tried to jack up the price substantially.

They've been playing these pricing games on and off for at least 10 years, if not longer. I have been willing to pay them at least $1,000 over that time. Instead they've gotten at most $200. But they're not alone in this. Most newspapers and magazines follow that pricing model.

OTOH the NY Times is great. I always clear cookies and browse w/o Javascript and I can read all the articles for free. This is a deliberate decision on their part. Still, I probably don't spend more than a few minutes a day on their site, so it's not like they could get $100/yr from me.

For a while, the NY Times was hilariously free. News articles were free, while they charged for their "editorial" content. That was perhaps a good idea for their core NYC demographic, but for other people (like me) the opposite would have been more attractive. How much would it cost me to keep from ever even seeing a headline for anything written by Maureen Dowd? Alas, that pricing model has long since changed.

I think you had a branding problem. Yours appeared to be an attack on a brand that is doing good work; this one seems more to be thwarting a sinister publisher practice.

Responding to specific criticism from your comments:

> It does feel excessive or cynical or something to install an actual app to work around a paywall. The WSJ organization says, you must pay us $28.99 a month for full access to this content. That's the value they've placed on their content, that's the price they are charging, and that's what people should pay.

> The fact that they also make most if not all of the content available on a "complimentary" basis via Google suggests that they do value occasional visits from non-subscribers.

> However if millions of non-subscribers were to install a paywall avoidance app and access the full content on a daily basis, this would probably violate the spirit of "occasional".

WSJ certainly deserves to set whatever price on their work that they wish, and they also deserved to be compensated according to the value of that work.

However, if the workaround is no more in violation of the "spirit of occasional [visits]" than this publisher practice is in violation of the "spirit" of web navigation. That spirit is that users should not be discriminated against based on their navigation source.

Google search results should not be a representation of content that is not allowed to direct visitors. If WSJ wants to represent to Googlebot and Google visitors that the content is open, then it should be open to direct visitors as well.

If I, a user, did nothing special to access the content, I should be able to pass on the link to a new set of users. This practice breaks that, and, I imagine almost invariably, users will not understand why.

> WSJ certainly deserves to set whatever price on their work that they wish, and they also deserved to be compensated according to the value of that work.

Don't you think we should also have the option to decide not to pay using these "dirty" tricks of the trade?

I think we should (and do) have the option to decide not to pay and not consume the content.

That's an answer to a different question. If WSJ, an amoral business entity, decides to serve different content to Google users and different content to everyone else, then you're left answering a question about user intent.

Personally, I don't think it's fair that WSJ simultaneously demands money for content but leaves open a loophole so they can get Google exposure, and then complain when people use said exposure. It smacks of wanting to have their cake and eat it too.

The entire point of the Google link and the whole paywall circumvention is so they can get traffic to their site, right? This implies that getting people on their site is more valuable to them than locking their content behind a paywall.

They could always serve only a summary to both Google and to nonsubscribers from other referrers, but this leads to people mentally filing them in the "useless link, never click" bucket along with Experts Exchange and other crappy content farms.

Actually, Google requires free article access from referrals to be included in Google News at all. (There are other ways, but I think that one was recommended.)

Yes, that is what I am trying to say. Karunamon (above/below[1]) said it better than I did. The WSJ deserves to be compensated for their work if they ask to be (in other words, consumers should not "steal" the content), but my questions is whether they have actually asked for the payment.

In my opinion, this tactic is no different than offering it for free but then demanding payment anyway.

[1]: https://news.ycombinator.com/item?id=9534392

Disappointed in you Jeremy.

Its fair for many here to complain about some of the online advertising practices of publishers.

However the viable alternative it to pivot to some form of subscription model. Its a hard transition and many orgs are making risky changes to do better.

Extensions like these are a step backwards.

If you value the work you should support it.

See bio for usual disclaimers

Can you share any feedback on the demographics of subscribers who successfully convert via paywalls? I'm curious how well it works with the 18-25 and 26-35 demos. Obviously this is a bit of an echo chamber here but I think a lot of us become annoyed and mentally boycott those sites instead of succumbing to pressure and paying.

Have you ever considered doing cross-promotions through other channels, such as the ability to round-up my Starbucks purchases and have the excess go toward a subscription?

How do you feel about Adblock?

Not a fan of the shaming...

If news orgs value their journalists and the future of the profession, they should experiment with more daring models, rather than ones that simply maximally enrich the org under the same old model

EDIT: imho a model that falls apart legally under a browser extension that fits in a 40-line gist is nothing on which to base the livelihoods of a bunch of people

I agree - but to be fair you're assuming they haven't experimented. Micro-payments, dumb 'capthcha' based models, memberships, events, "premier access". All have been tried in various flavors. Some with limited success.

I'm not asking for you to dream up an alternative solution, but don't assume this is a problem that has not been given a lot of thought by many organizations.

This is partly why we are stuck with things like "native advertising" and pretty shit IAB ad units.

Even people inside the NYT know the pricing for digital subscriptions is excessive. They're high because circ department won an internal war and inflated digital prices in a misguided attempt to protect print subs. The NYT has experimented but is still clinging to the past.

Which is why the "All Digital Access" subscription costs more than Netflix, Hulu, and HBO Now combined. Give me a break.

Almost thought you were joking, but wow... nearly $38/mo to read NYT's webpages.

This raised an interesting question for me - how much money is saved on a per-subscriber basis by sending the digital version vs. print (for the NYT)? Their pricing certainly doesn't reflect it:

  Web + Smartphone:    $3.75/wk  $16.25/mo  $195/yr
  Web + Tablet:        $5.00/wk  $21.67/mo  $260/yr
  All Digital Access:  $8.75/wk  $37.92/mo  $455/yr
  7-Day Home Delivery: $8.90/wk  $38.57/mo  $463/yr

There are multiple levels of insanity here:

  Web + Smartphone & Web + Tablet are identical except
    for device type but Tablet is 33% more.
  Web + Smartphone plus Web + Tablet cost as much
    *together* as All Digital Access (and are feature
    equivalent).
  7-Day Home Delivery *includes* All Digital Access and
    costs about the same. Paper is free?

None of these pricing strategies appear to be designed to maximize readership, broaden subscription basis, or grow brand penetration. They are structured to capture the most value possible from customers with high switching costs or price indifference. This is what a conservator in a dying market looks like.

Actually, thats not true. A broad amount of research wen tin to this (originally at least).

There is a method to this madness :)

Please tell us more!

The reason that print+web packages don't cost significantly more than web-only is because print advertising is still a lucrative business. NYT can a a absolutely justify giving away the paper to subscribers on that basis.

Except, if that were true, why not just make it a free paper? Presumably ad targeting... I understand the limitations of their existing revenue model. It just results in some pretty consumer-unfriendly offerings, at the expense of readership and penetration...

---

EDIT: Went back to look at the numbers... They have tons of online eyeballs - those eyeballs just don't make much money. They manage to double their digital revenue with the paywall. Even so, it's dwarfed by the print revenue.

This may shift as advertisers see the value of targeted online user groups... I could totally imagine the NYT of the future being a collection of blogs dealing in Tech, Biz, World News, Local News, etc. Ads on the Biz site would be targeted differently than on the World News site. This might give advertisers what they want in terms of a differentiated audience.

---

From the 2014 Annual Report:

  Average Paper Circulation in U.S. (Int'l)
    648,900 (219,500) for weekday (Monday to Friday)
  1,185,400 (220,500) for Sunday

  Paid Digital Subscribers
  910,000 as of December 28, 2014

  Monthly (Annual) Digital Readership
  31MM (372MM) unique desktop/laptop visitors (U.S.)
  42MM (504MM) unique desktop/laptop visitors (WW)
  28MM (336MM) unique mobile device visitors (U.S.)

Off these eyeballs, they made:

  $667.5MM from print subscribers
  $169.3MM from digital subscribers

  $483.5MM from print advertising
  $178.8MM from digital advertising

On the smart phone, you don't see the comments for an article but the Tablet version gives you the comments. I can't say if the comments are worth the 33% extra cost but I typically end up reading the web version because I find the comments on the article usually interesting. I ended up going for the Web + Smartphone version because I like reading NYTimes and this was the cheapest I could afford. I've also found quite a few people in this same position.

   The NYT has experimented but is still clinging to the past.

Yes and No. Print advertising is significantly worth more than web. Thats not "in the past' that is alive and well (but declining). Still high.

Ya but then isn't it pretty logical that people use a browser extension to get around buying subscriptions that are priced artificially high to protect a soon-to-be-obsolete revenue stream?

You're right. I was hoping to clear up my support of his extension with the edit, but it was still lacking. I realize I'm a peanut-gallerian, and both of you two are working harder than I on this.

I support him because it shows me there's someone inside who's pragmatic and not letting themselves be deluded by group-think. (I am genuintely not implying you are. I don't know you.)

Paywalls aren't going to work as-is, and I'm excited to see a clever wink to that sentiment internally -- it gives me confidence that something better might come from inside players

> I'm not asking for you to dream up an alternative solution

And not my idea, but just in case it's not on your radar: http://wiki.bitshares.org/index.php/Potential_DAC_Ideas#Asso...

When have micro-payments actually been used in any meaningful way? I've only seen companies complaining that they aren't viable. They were certainly quick to make the choice for me.

Otherwise, if people saw truly unique content from you (and others), they would be paying in one or more of the options you mentioned. They obviously don't see the point, however, because very little on the web is truly unique.

This seems like more evidence of the thing we all already know... large news/opinion organizations, especially those who started in print and are trying to maintain their business models, have become irrelevant.

   ...have become irrelevant

Please read this article:

"Cuomo Orders Emergency Measures to Protect Workers at Nail Salons" http://www.nytimes.com/2015/05/11/nyregion/cuomo-orders-emer...

That response by a state governor was in due to an article in The New York Times from a week ago.

Also, do not underestimate the amount of original content that these orgs churn out that other sites repackage and repurpose.

Irrelevant? Do some measure of homework.

I think it's wonderful that the NYT made a difference, but that certainly doesn't prove your point.

> original content that these orgs churn out that other sites repackage and repurpose.

What percentage of their content are truly original, first-to-break stories? Really, be honest. Now how much of their content are fluff/opinion pieces, rehashes of the last month of stories, and info that has been 'repackaged and repurposed' from the internet? If you try to claim the larger percentage is the former, you're just being obtuse.

You're quick to defend traditional media, but you totally glossed over the other points I raised. It's obvious (based on your comments here) that you have a bone to pick with anyone who doesn't believe your employer (and others like it) are worth what they think they are worth.

Again, it's clear based on market force that the PUBLIC doesn't agree with you. If they did, you would have seen success in the methods you mentioned.

> Micro-payments, dumb 'capthcha' based models, memberships, events, "premier access"

Whether you personally agree with me or not, the public is speaking, and they are telling you that your business model isn't worth what you claim.

It seems like you're arguing about different things. donohoe took issue with you saying the NYT isn't relevant, and you are stating that people's unwillingness to pay equates with irrelevance.

I don't buy that. Many new media sites don't even have reporters in the field - just lots of writers in a room that rewrite what these "legacy" orgs do. So if they disappeared people absolutely would notice, the connection just isn't apparent yet.

News costs money. And a functioning media is important to society. That people are not currently willing to pay for it doesn't mean that it's irrelevant, just that the correct business model has not yet been found.

For the record, my original comment never mentioned the NYT. I said, "large news/opinion organizations, especially those who started in print and are trying to maintain their business models, have become irrelevant." donohoe responded with an article in the NYT that caused the NY governor to take action on something. At best, that illustrates that they wrote a good article. It does nothing to provide an answer to my original comment. So yes, I argued one thing, and he argued something totally different so that he wouldn't have to answer the tougher question.

> That people are not currently willing to pay for it doesn't mean that it's irrelevant, just that the correct business model has not yet been found.

Just like the MPAA and RIAA a decade ago, they have been fighting market pressure rather than taking the hint and changing their business models. They want to keep the same profits they received from their captive audiences of yesterday, and they don't understand that people don't need to pay historical rates for the vast majority of the information they offer.

If the NYT fits that description, then I would love to see a reason why my conclusion doesn't apply to them. They certainly aren't relevant because of their name alone, are they? So I would truly like to know what percentage of their content comes from the real NYT reporters in the field, versus internet research, AP feeds, rehashes of yesterday's news, etc.

Joe Q. Public certainly doesn't miss out when he avoids paywalls. He typically gets better (and faster) information through his RSS feed than he ever has by reading the NYT directly. They might offer a unique opinion, or an extra line from an interview, but it's rarely vital information. He's not going to pay monthly for the privilege of reading an article written by a 'distinguished journalist' when it's really all about the information, not the prose. So what does that leave? Exclusive photos? Infographics? Op-ed?

I really don't mean to imply that journalists who go into dangerous situations aren't worth being paid. They certainly are doing important work. But to imply that the ones employed by one news corporation are any better than the others is just silly. They are all witnessing the same event from different angles, and I now have the ability to see the situation from the eyes of ALL of them through multiple sites (often within hours), not just the one who graces my newspaper's front page every morning.

So in the end, they have built a business on being a person's sole channel for world news, but the average person has many many many more channels available today. If they don't change their business model (indeed, their entire way of thinking about news), they will become unnecessary, extraneous, pointless... irrelevant.

This does not come across as very well-informed. You are replying to someone who is talking about the NYT, which, whatever its limitations, is probably the best daily source of the thing we call "news" in the U.S.

You say that there is little unique content out there -- but this a false statement if we're talking about the WSJ and NYT. If you are talking about the web as a whole, then you've changed the subject, and you're talking about an unbounded thing.

Saying the NYT is irrelevant is also completely false. Endangered? Fearful? Sure. A mouthpiece for the establishment? Often. Caught in their own thought-bubble? Sometimes. But not irrelevant.

Information and quality analysis have value - this is obvious. Coming around the other way and saying that because publishers are failing to extract the value, that therefore the news has no value, is a fallacy. Markets can fail to price correctly for many reasons.

Clay Shirky's original 2009 essay on "Thinking the unthinkable" (http://www.shirky.com/weblog/2009/03/newspapers-and-thinking...) was plenty frightening for publishers, and it did not need to exaggerate to make the point.

The model doesn't fall apart legally because it can be trivially circumvented. Any more than the model of renting bikes falls apart legally if people just don't bring the bikes back, or the model of selling software licenses falls apart legally because people can easily copy it. You can base the livelihoods of people on these things, provided the amount of circumvention remains low.

I see what you're saying, but I wasn't intending to stretch my thought that far. I was just assuming that changing the referral header couldn't possibly be illegal.

More daring models? It's easy to just say "do something else" when actively circumventing a system. I really wish people would generally stop with comments like this unless they have an alternative to propose, otherwise its just criticism without purpose because the people involved in this industry are trying their best to make things work.

What does the size of the "hack" have to do with anything? Simple execution doesn't make something ok by the fact of being simple to execute.

Also, not my intent to shame. And I'll apologize to Jeremy next time I see him if it came across that way. Actually, as I write and re-read it, it does come across that way.

Sorry.

If it's really so concerning, then perhaps the technology should be made a bit more difficult to thwart. This extension is simply changing the Referer request header.

Sorry to hear that, Michael.

Subscription models are great, and paywalls are great.

What's not so great is creating an internet where if you use our Trusted Corporate Partner™ you're allowed to view a public webpage, and if you come from anywhere less privileged on the internet, you're not. That's not sustainable. Lest we fall into a vortex where you're only allowed to browse the public web by logging in through Facebook. I wouldn't want to be online in that world.

This isn't a complicated extension, and it doesn't do anything fancy to subvert any paywalls — and it's not going to. It simply reloads the page as if you had arrived via Google. I think that's a fair thing to do.

  ... if you use our Trusted Corporate Partner™

No. Not like that. Basically, its Google saying publishes must do this or they won't send traffic their way (via search result / SEO)

Its not publishers saying use Google over Bing over Yahoo etc.

  Lest we fall into a vortex where you're only allowed 
  to browse the public web by logging in through Facebook

The other argument that this may hold water is similar exceptions when you come from social (Reddit, Twitter, FB, etc) where they often allow access. However not for the reasons you're suggesting. In that case its the publisher saying they don't want to cut them-self out of the conversation (who'd link to the NYT if it was alway a paywall or would immediately count against your monthly free views before you can determine whether you wanted to read it or not...). In such cases those exceptions are factored into the economics of whatever the 'free views' limit is.

  and it doesn't do anything fancy to subvert any 
  paywalls — and it's not going to

Really? You could make that argument. However the screenshots you choose for the extension in the WebStore are all sites that have paywalls or subscription requirements. So that last statement is being a big disingenuous.

   I think that's a fair thing to do

Makes sense from google's perspective. Why should they list sites that people can't access.

Listing sites people cannot access does sound like a bad user experience.

Who'd link to the NYT if it was always a paywall, indeed?

Of course I understand that the publishers don't want to cut themselves out of the conversation.

But why must "the conversation" be limited to powerful internet companies? Do we not converse about news articles in blog posts? Comment threads? Up and coming social platforms that don't have Facebooks media clout yet?

It's simply unfair to treat a visitor arriving at your site from Google differently to a visitor arriving at your site from anywhere else. It runs against the grain of the web. To that end, if I hadn't published this extension, someone else would have made it. If Google removes "Referer" privileges from Chrome extensions, someone will fork Chromium to allow them again. Let's build a better — a more fair — paywall.

>Let's build a better paywall.

and hasten the death of print journalism

I'm honestly fine with that.

Yes, please.

So publishers should either block traffic from search engines, or not have a paywall at all. Neither seems economically viable to me.

It would not be hard for Google to append one-time, time-limited tokens to SERP links to get searchers from Google to NYT (or other big publishers). Google would just need to provide an invite-only API for the publishers to check the tokens against in real time. At their scale this would be pretty trivial. This would have the neat effect of breaking your extension, and locking in special treatment.

Google would not bother to do this right now, because the referrer trick largely benefits Google (the typical advice is "just Google the headline"), and is not widely abused. But if people start widely circumventing both Google and the paywall (with something like your extension), then the interests will align and Google and the publishers will act together.

Google executives have said repeatedly and publicly that they want existing news organizations to survive and feel a responsibility to help them do so.

So your proposal if to make Google say "hmm, I don't really think you searched for that, so the link is going to have missing content"? Otherwise how are they gonna differentiate between someone searching and finding the article "by accident" vs on purpose? Sounds like a crappy user experience.

> No. Not like that. Basically, its Google saying publishes must do this or they won't send traffic their way (via search result / SEO)

Links published --anywhere, not just Google-- for content behind paywalls are called ads.

I think the real "issue" here is that those paywalled content publishers want to get paid but also want the free advertising that comes with having a well indexed site.

It's a pretty interesting subject in my opinion because I can easily see arguments for both sides, and if we compare it to the real world we have the exact same "issue.

I can read an article that's on the newspaper's first page when it sits in the newsstand but I can't open it and look for any other articles without paying for the paper.

I think the real solution has to come in the form of a better subscription model. I usually get to paywalled sites wanting to read a single article so a subscription is a bit too much for me. If there was a way for me to get what I want by paying quickly, maybe using some sort of a federated payment platform (much like downloading apps doesn't require subscribing) I'd be more inclined to pay.

>> "If there was a way for me to get what I want by paying quickly, maybe using some sort of a federated payment platform (much like downloading apps doesn't require subscribing) I'd be more inclined to pay."

Isn't this what Flattr tried (and failed) to do?

Flattr is more about donations, not payments. As far as I know, you can't detect if the person has flattred your page.

It's quite a bit absurd to complain about technical solutions which fullfill the needs of users. If a source of income revolves around annoying users into paying and for them not to find a way around such annoyances, don't expect it to last forever.

In this model, I consider the whole tactic to be publishers relying too much on Google's policy in the first place. If they change that policy, said publishers are screwed. Anything that depends on a third party is doomed to begin with.

These things are temporary income at best. Publishers should not just "take the win and go home"; keep looking for other, better solutions.

> It's quite a bit absurd to complain about technical solutions which fullfill the needs of users.

Nonsense. Users want everything for free, instantly and with no ads.

I would like my sandwiches from the shop downstairs to be free all the time. But it's not cool to forge stamps on a loyalty card.

> Users want everything for free, instantly and with no ads.

And where's the source to that little strawman of yours?

I'm sure users also want hot threesomes and a million dollars.

I think we're perhaps talking past each other. You want a source that says people prefer getting things for free over paying for them?

Life and business is all about 3rd parties and working with and against them. It's quite easy to say that things should just be better but it's not like all these publishers are stupid and just sitting around not trying to come up with better solutions. This is actually the best we have at this moment without a massive shift in how the web works.

The fact is that the internet makes things very easy to take without repercussions. Physically walking into a store and taking something has consequences but a quick hack on the browser is basically unnoticeable. The abilities and democracy of the internet architecture facilitates the extraction of content without compensation. Just because it's easy though, doesn't make it "right".

There's no taking going on here. Taking something implies a transfer of possession, leaving one party no longer in possession of that thing. My browsing a web page does not result in such an outcome.

Taking is not zero-sum. Virtual existence does not disqualify ownership or rights. Content is intangible but still carries value, the same way millions of people can have a copy of a song but it is still property.

Receiving that value in the form of content without compensation, either by viewing ads or paying money, is still "taking" it against the wishes of the provider.

Reminds me of the very old joke about software:

   Software is like VD
   you can give it away and you've still got it

> Just because it's easy though, doesn't make it "right".

If something is both easy and rewarding for those who do it, don't base your business on trusting people not to do it.

Consequentially, don't base your morality code on a notion of "people shouldn't do it". They shouldn't do it because it hurts businesses, but it only hurts businesses because they're foolish enough to make the first mistake.

I believe your foundation for this argument is flawed and doesn't answer my statement: just because something is easy, does not make it right.

"People shouldn't do it" is the fundamental way laws and regulations work. You shouldn't damage property or steal items or assault people. But you can. Very easily. And depending on when/where/how you can probably get away with it. Does that make it ok?

Actions in the physical world have very lasting effects and, these days, leave a big trail which allows the proper authorities to catch the bad actor and dispense justice (the topic of what is proper justice being out of scope for this thread). The digital world is much harder with things happening instantaneously with very little if any trail. If publishers could catch you, they would. Realistically, they just can't. But this lapse in their abilities does not empower and justify your abilities to do what should not be done.

Business models are a completely separate topic and, yes, it's usually good to align the model with normal consumer action. However, human nature is to always take with the path of least resistance and in the case of "value", to acquire it for nothing in exchange if possible. Most businesses work because they can block any access to that value without a proper exchange. Internet publishing however is very unique in that the very structure of the web which makes it so open is also what makes it impossible to stop and effectively filter anyone. The paywall is one such filter but they have this opening for access to Google listings and let a few users be randomly awarded free content. If there was a way they could be listed on Google without losing their paywall, they would, and this whole situation would be non-existent.

But just because something hurts a business doesn't make it wrong, either. And it's worth considering that businesses are amoral entities - given the chance, most of them will gladly screw you if it increases their revenue.

With that in mind, why should a rational human choose to apply morality to an entity that has no obligation to do the same in return?

(This is not a value judgement on WSJ in particular, more a general statement as to the way of the world)

The way I see it, there are two arguments for this plugin to not exist:

1) "What if everyone does it? If everyone does it, WSJ makes no money and goes away". The appeal to utility. Problem is, it carries the unstated and unsubstantiated assumption that enough people will use this plugin to actually impact their profits in a measurable way.

Because of that, I call this argument invalid.

2) "It goes against the wishes of the content providers". The appeal to morality - but as mentioned above, this is strictly a one way consideration, and it's subjective for each person. There's also the moral consideration that WSJ has chosen to make this content visible for Google users, but only because Google forbids showing different content to the spider and to the human. Then, you're left answering an objectively unanswerable question - is a person who googles for most of their news and ends up on the WSJ often doing something wrong? Or does it only apply when you install the plugin and circumvent the company's intent?

The net effect on WSJ's profit is the same either way.

It's not about hurting the business. It's not just "wishes". That's the legal framework, also laid out in their terms of service.

You're breaking the contract that the content "value" is provided in exchange for something, either attention on ads or cash, depending on the publisher. This is what makes it wrong.

The real issue is that the internet does not allow for effective policing and capture of contract offenders. It's wrongs being committed with no consequence, but this does not justify it being done.

But if I'm on their site reading their content and linking to their other content and looking at their ads, how am I not fulfilling that "contract"? Attention is what they wanted, attention is what they got.

That contract I never was given a chance to read, or sign, or agree to in any way, mind. Given that, i'm not sure how you can say refusing to do something I never agreed to do in the first place is morally wrong.

This is my whole point - the distinction is entirely arbitrary because WSJ obviously values the attention more than they value the money, else we would be having this conversation right now.

This is a positive thing has been made, and I too have worked at publishers. You can't control these things and this tool prevents a user from having to jump through hoops to use a feature which is publicly available. Well done. You want your geek customers to have some chance of once-again enjoying reading these sites so let them back on them.

No, you can't control these things. All I'm saying is I'm disappointed to see yet another one.

However I would much rather spend my time building new features then having to find ways to ensure a paywall is not circumvented.

This extension simply modifies the referrer, overcoming some strange problem with certain websites where it alters page content depending on the referrer header - i.e. it fixes URL uniformity on broken websites.

I'd be all for this extension not needing to exist if these website owners would simply fix this obviously broken behavior.

I'm a bit curious. I was under the impression that changing your site to show different content to "Google" than "regular users" was against TOS and could get you banned from results all together. Do big companies pay to get around these restrictions or have there been rule changes in Google Search?

Google is OK with it as long as you show the same content to a user going through Search as you do to the bot. They don't care if you show different things to users coming through other sources or directly hitting the URL.

I see you wrote a Chrome extension that cuts most of the ads from NYT.

Yes. My guess is you're leaving that statement hanging to make a larger point?

If so I'd love to add that I'm annoyed by some of the tactics and lengths that publishers go to to monetize content ad the expense of user-experience and other costs.

The only path right now are ads, 'sponsored content', 'native units'.

I would much rather we have a subscription-based model that generated enough revenue to ween them off awful ad experiences.

Thats very much an aspirational goal.

My larger point is that having readers circumvent that, as opposed to supporting that, is disheartening to many of us who are working on the inside to push change.

BTW: And that extension was a redesign, of which ads were removed or moved. The NYT legal dept had Google remove it.

I support subscription content. But you can't have your cake and eat it too -- in other words, you can't put content behind a paywall and also have it publicly indexed by search engines.

Remove the trick where it's okay for Google searches to circumvent the paywall and this extension becomes useless.

Why not? So paywall content is not good enough? It's just a different business model by publishers: free access with ads or pay for access.

Why does that penalize the content itself and the discovery of it?

What's happening here is that it's not just "free access with ads or pay for access." It's "free access if you came from google, so that google can index our pages, but otherwise pay for access so that we can put the same content behind a paywall." It's less a business model and more an SEO trick that Google puts up with under certain conditions.

I agree with most of your other posts on this thread but why is it ok to just remove ads from other competing publishers?

It's their choice to choose a business model, whether ads or subscriptions. If they serve ads and it's not the experience you want, shouldn't you just move on and not go back to that site?

The ad model actually works well for lots of sites and is the sole reason we have the long-tail of the internet. Someone working at an ad-based publisher would have the exact same comments about your ad disabling extension, why circumvent if they're trying to make it work?

> I'm annoyed by some of the tactics and lengths that publishers go to to monetize content

> having readers circumvent that, as opposed to supporting that, is disheartening to many of us who are working on the inside to push change.

So what you're saying is that it's OK to circumvent the methods of monetization you don't support?

He doesn't like paywalls, so he circumvents them. You don't like ads, so you circumvent them. And then you say shame on you? This is absolutely the pot calling the kettle black.

Why should I support the work? The work is given away for free on Google searches. I value a WSJ article very little unless it is free. I want to support their work with exactly zero dollars. Why is that a problem? I would prefer that everyone working at the WSJ were just doing something else with their lives. Is it okay for me to align my spending with my values?

That's a tenable position, but this extension should not be useful to people holding that position.

Guess what I do when I hit a paywall like this? I get annoyed and close the site. Just because I click a link doesn't mean I value it. I would be just as happy if no one ever sent these links to anyone.

As far as I'm concerned people that let google index their site and try to paywall everyone else are the ones abusing the system.

Exactly what I do too. I suspect it's pretty common. I've never felt that I missed out because I didn't read a paywalled article. If anything, I probably found more interesting info in the comments.

I used to be a freelance writer. I love writing. I wrote for several magazines, a daily, and a weekly. I even consulted at a newspaper back in the 80s installing desktop publishing capabilities.

I love printed media.

Printed media is also dead.

I wish this were not the case, and I know lots of folks are trying really hard to prevent its complete destruction, but heck if I can see a way forward.

Google is not the magic gateway to the internet, and sites that treat it like it is will have this happen to them, just like the sun follows rain.

I do not enjoy pointing these facts out, and I worry tremendously about the quality of journalism, writing, and online discourse. But the facts are the facts. This has been a losing battle long enough for all of us to figure out how it ends.

The paywall model is currently broken:

* not culturally integrated the way we don't mind paying for smartphone apps;

* not atomic enough (we want to pay cents for an article, not dollars for a website, nor tens of dollars for a long-term subscription);

* it raises convenience and trust issues;

* it competes against ad-based models, which adblockers have failed to bankrupt until now;

* paywall sites aren't taken into account properly by Google et al.

Relieving pressure from the currently broken paywall model won't help fix it. Making the situation untenable is the only way to make it change. The most effective move would be to kill ads as a business model. Of course, don't expect Google, an advertising agency, to help us do that. And how to pay author while keeping Internet searchable remains an open problem AFAIK.

we want to pay cents for an article, not dollars for a website, nor tens of dollars for a long-term subscription

Is that true, though? I think Spotify has proved that people prefer subscriptions to one-off payments for music, for example. Having to make a purchase/don't purchase decision for every news article you read doesn't seem like a good experience at all.

Yes it is true. For a single payment to Spotify you get access to music from all four major record labels.

OTOH paying for a subscription to a single newspaper is like subscribing separately to Sony, EMI, Warner Music Group, and Universal. Not gonna happen.

I think they would be much better off with the Spotify model. I would happily pay a monthly fee for a collection of all the major newspapers.

That depends on the price.

I just checked out WSJ and NY Times. After promo, WSJ is $29/month. That's whether or not you want actual paper or just digital access. NY Times has various plans, one of the cheapest is about $16/month. So that's just two newspapers.

I'd gladly pay $20 or even $30 per month for full access to all the major newspapers. But the big papers would never agree to that, because their individual cuts of the pie would probably be too small.

And we are disappointed in your crappy paywall model.

how would you feel about an extension that simply removed hyperlinks to paywalled pages, or highlighted them in a different colour? (serious question; I've been tempted to make one simply because I feel that publishers are welcome to paywall their sites, but then I don't want them on "my" web.)

Problem is that Google, HN, "share" links etc is free advertising. Media says that certain discovery channels (Google) should be let in, while other (HN, emailed links, etc) shouldn't. This extension only equalizes that.

Perhaps a better extension would be to hide organic search results in Google for pay-walled content providers, while leaving AdWords links intact. In the interests of protecting profit-based interests.

Additionally, Facebook should require paywalled content providers to pay users that use their integrated comments or click a like button, as we certainly don't want free buzz, as that would violate the profit-driven flow of information.

Publishers are all for-profit, they just choose different monetization models. Whether it's ads or paywall, content is content.

Why should it be ok for Google to discriminate which content is shown by the host's business model instead of the relevance and quality of the content?

I'm cool with content providers charging. What I'm not ok with is some discovery channels getting free access to that content, while others don't.

As far as Google is concerned, they're a business, not a public utility. If a content provider can choose what and whom to show to, so can Google.

The ethical approach would be to not be indexed by Google, but rather, use a pure paid ads approach (or a revenue sharing model).

To not have a universal interaction model is deceptive. (I search on Google, see an article, click, and use your nifty sharing widget, but those I share with are prompted to subscribe, and now I look bad)

The free-for-Google paywall is in many ways similar to online advertising: it's all bait-and-switch.

Anything that discourages such practices is a step forward.

It's time publishers start considering their readers as customers to be treated with respect instead of either dumb cattle or the enemy in a digital arms race.

What kind of a legit, respectable business tries to trick people into thinking their giving their product away for free? At the very least don't whine about the public tricking you back in return.

Great idea. This will probably be up there with the invaluable-ness of Adblock. I have nothing against companies like the Wall Street Journal who want to charge visitors to see content, but if you are going to paywall content at least be consistent and charge everyone or don't charge at all. You can't give it away to visitors from Google and expect people who weren't lucky enough to come from Google to have to pay. It is unfair and discriminatory in my opinion.

This kind of reminds me of the aged old trick that people used to use for Google to get better SEO. Normal visitors would get one version of the site, the Googlebot would get an over-SEO optimised version of the site stuffed with keywords and junk (I am talking early 2000's here before Google was as advanced as it is now) to give the impression of a different site.

Like I said, I think content publishers should be able to charge people for their content, but you can't have two sets of rules for the same content for the sake of a better SEO ranking and NOT expect people to exploit that loophole. What WSJ and other content publishers are doing is a little borderline dodgy.

I am pretty disappointed in the comments from some people here shaming the author for creating this plugin. He isn't doing anything that we haven't already been able to do for a while now. He isn't breaking into servers and getting the content for free, he is exploiting an SEO trick WSJ and other sites are using to avoid being penalised in Google. Completely legitimate if you ask me. If WSJ wanted to truly protect their content behind a paywall, they would have presented the wall to everyone regardless of their referrer.

You can. It's a form of price discrimination. Like coupons in a supermarket. People who really want to pay less will jump through the hoop. If you just want it conveniently and don't care about the cost, you can subsidise those others. In a way this is more fair - it means people pay closer to what the money is worth for them.

IMHO rendering totally different things depending on the http referrer should be considered as cloaking. It's basically the same practice than rendering a different thing if the User-Agent matches Googlebot.

I wonder what would happen to your site if you did this without being The New York Times.

Kudos to Jeremy for this.

Google actually encourages this and call it first click free.

I think it's funny that Jeremy made this considering his background.

yeah i came here to say that

I was curious so I googled him and saw he created Coffeescript, backbone.js and underscore.js. Is there part of his background I'm missing that makes this funny. Seriously I don't get it and it's bugging me :)

The backbone.js stuff was during NYT work if I recall.

Ah ok, that is pretty funny.

That's not quite correct. Both Backbone.js and Underscore.js were extracted from DocumentCloud — a nonprofit organization.

But I am currently a Graphics Editor at the NYT...

Not for much longer, I bet...

Ha ha... We'll find out soon, won't we.

Is he not still at the New York Times?

He is

The person who build the paywall is the best qualified to tear it down. :)

This is gonna mess up some major web analytics stats about referrers :)

As someone who works with SEO, this was my first reaction. It's going to make a lot of my analytics wonky and the numbers won't be very accurate.

I guess you have to assume such a small percentage of people will actually be using this, so the numbers won't be too bad??

As someone who works with SEO, maybe it's time you learn your entire field of work depends on the good will of both search engines and users to not put a screwdriver in the ball you play with.

Actually, I do more development work than SEO, so referring to my "entire field of work" is a bit. . .well. . .erroneous.

But hey, whatever makes you feel better pal.

I guess you have to assume such a small percentage of people will actually be using this, so the numbers won't be too bad??

You don't "have to" assume that, but why start worrying about the source of error in your metrics now?

The cumulative wight of this and all the other problems with your data probably won't dip their utility below zero, so just go back to plugging your ears and yelling "lalalalala".

It's the same thing as with browser identification and detection of mobile devices - as soon as you start using them for non-info purposes e.g. disabling content or redirecting people to a mobile-specific page (or an advert for 'app of your site) that they don't want, then you have to expect that any well-meaning software will have the user device lie to you, because it's in the best interests of the user.

It only changes the referrer after you click the button. You have to be sitting on the page first.

Only if a lot of people use it. I suspect something like this can continue to fly under the radar while only a tech savvy minority use it.

I'd rather have a plugin that just ensured I never went to these sites at all. Or a list to drop in my hosts file.

While I don't have a list of paywall sites, I've been compiling one of Weaponized Viral Clickbait sites generally and the Spartz network (see the New Yorker's profile of the "Virologist" from a few months back) specifically.

This is in addition to a set of blocklists curated from the uMatrix Chrome extension.

    0.0.0.0			knowd.com		# WVC
    0.0.0.0			www.greenmedinfo.com	# WVC
    0.0.0.0			www.heavy.com		# WVC
    0.0.0.0			freepatriot.org		# WVC
    0.0.0.0			www.uproxx.com		# WVC
    0.0.0.0			tcr.tynt.com		# WVC
    0.0.0.0			slightlyviral.com	# WVC
    0.0.0.0			pro.moneymappress.com	# WVC
    0.0.0.0			pro1.moneymappress.com	# WVC
    0.0.0.0			weknowmemes.com		# WVC
    0.0.0.0			www.shareable.net	# WVC
    0.0.0.0			www.thisblewmymind.com	# WVC
    0.0.0.0			www.tsu.co		# WVC
    0.0.0.0			www.upworthy.com	# WVC
    0.0.0.0			www.distractify.com	# WVC
    0.0.0.0			news.distractify.com	# WVC
    0.0.0.0			www.evwow.com		# WVC
    0.0.0.0			evwow.com		# WVC
    0.0.0.0			www.wimp.com		# WVC
    0.0.0.0			livelol.me		# WVC
    0.0.0.0			www.scoopwhoop.com	# WVC
    0.0.0.0			bustle.com		# WVC http://www.bustle.com/articles/62365-there-are-no-words-for-how-horrifying-this-unknown-creature-found-in-a-tuna-can-is
    0.0.0.0			www.bustle.com		# WVC
    0.0.0.0			www.tickld.com		# WVC
    0.0.0.0			www.zmescience.com 	# WVC
    0.0.0.0			www.addictinginfo.org 	# WVC
    0.0.0.0			www.viralnova.com	# WVC
    0.0.0.0			opter.co		# WVC
    0.0.0.0			www.sliptalk.com	# WVC https://plus.google.com/u/1/+IrreverentMonk/posts/cuNpUdErc7V
    0.0.0.0			sliptalk.com		# WVC
    # Spartz:  http://www.newyorker.com/magazine/2015/01/05/virologist
    0.0.0.0			www.brainwreck.com	# Spartz network
    0.0.0.0			www.dose.com		# Spartz network
    0.0.0.0			www.unfriendable.com	# Spartz network
    0.0.0.0			www.smartphowned.com	# Spartz network
    0.0.0.0			smartphowned.com	# Spartz network
    0.0.0.0			www.givesmehope.com	# Spartz network
    0.0.0.0			givesmehope.com		# Spartz network
    0.0.0.0			www.philosoraptors.com	# Spartz network
    0.0.0.0			philosoraptors.com	# Spartz network
    0.0.0.0			www.lolbrary.com        # Spartz network
    0.0.0.0			www.omgfacts.com        # Spartz network
    0.0.0.0			notsafeforwallet.net    # Spartz network
    0.0.0.0			sex.omgfacts.com        # Spartz network
    0.0.0.0			www.itsamememario.com   # Spartz network
    0.0.0.0			www.whenuseeit.com      # Spartz network
    0.0.0.0			memes.mugglenet.com     # Spartz network
    0.0.0.0			www.mugglenet.com       # Spartz network
    0.0.0.0			www.memeslanding.com    # Spartz network
    0.0.0.0			www.pokestache.com      # Spartz network
    0.0.0.0			www.ragestache.com      # Spartz network
    0.0.0.0			www.sixbillionsecrets.com # Spartz network
    0.0.0.0			smartphowned.hollywood.com # Spartz network
    0.0.0.0			www.hollywood.com       # Spartz network
    0.0.0.0			www.tasteofawesome.com  # Spartz network
    0.0.0.0			www.thatssotrue.com     # Spartz network
    0.0.0.0			unfriendable.hollywood.com # Spartz network
    0.0.0.0			love.givesmehope.com    # Spartz network

I need a 'facebook sent me' version, lots of websites allow facebook references but not neccesarily google.

I'm reduced to going to facebook and sending a message to myself with the url so I can click on it.

    > lots of websites allow facebook references but not neccesarily google

Like what?

Curious as well... Which sites?

Looks like most of them ALSO allow a google user-agent, testing with this plugin, so I'm good.

Mostly various newspaper sites.

Pretty sure it's easy to do since this one is open source.

Righteous indignation abounds, a good indicator of achievement.

Nice hack jashkenas.

I think they took it down :( guess I'm gonna have to find out how to set it up from source code: https://gist.github.com/jashkenas/9a4d9e648eb11aab394e

This seems like a feature which Privoxy could provide...

http://www.privoxy.org/

Been pulling something similar with Firefox and Modify Headers, only i claim i come from Twitter...

This extension seems to have been removed from the Chrome Web Store. Hmmmm.

Good idea, but does not work with Financial Times.

FT has (as far as I've been able to determine) a hard paywall. But they do give you a certain number of free articles per month (8 IIRC?) if you create an account. Also everything at FT Alphaville is free with an account.

Good enough for me!

Hi! I work for Financial Times.

We've started experimenting with First Click Free — soon you'll get one article a day for free if you come from Twitter, Facebook, LinkedIn or Reddit.

We'd like to let Google index more of our content (which will require allowing First Click Free from Google as well), but that depends on our other efforts, such as launch of a redesigned desktop site (you should try the beta, it's pretty cool) and technology changes behind mobile sites.

Of course we'll keep First Click Free if it leads to more readers subscribing rather than just faking referrers.

Can you add Drudgereport to that list? I follow a lot of links from drudge and when he links to a FT article and I click without realizing I end up at the paywall splash screen and just immediately close the tab.

So change useragent to googlebot?

No: https://news.ycombinator.com/item?id=9531943

Jeremy,

Useful plugin but BLAH chrome! Any chance you will port this to Firefox?

Google users are priviliged for a reason.

I thought Google users were only privileged because news sites want the full text of the page to show up in searches, and Google punishes pages that show different content to the web crawler than they show to searchers.

and the reason is: if a user bounces off a SEO result, the ranking is degraded, and nobody wants that, so Google users can skip the paywalls. I don't think that's a good reason, or even a good way to protect content.

Really? With a Google™ in the name, too? What do you think is going to happen in the next two hours?

1) They'll take down your extension because you're violating their ToS and their trademark

2) The extension will already piss off their "partners", but they can't really say that's the reason they're taking it down, so good thing you provided them with option 1)!

I have nothing against the extension itself. I search for such headlines all the time in Google, too. I just think it's reckless to taunt Google like this and give them an excuse to shut it down on a silver platter.

If you use a trademarked name you're supposed to put the tm there. e.g. I used F1 in the name of an app and was contacted by the rights holders. Their only demand was that anywhere I use the text 'F1' I put the tm mark.

I don't see how he is violating the trademark (though that ™ should be a ®).

It seems like Google recommends using ™ for most of their trademarks. Their website says that laws about using ® vary from country to country and there are sometimes harsh penalties for using it with an unregistered trademark. It seems at least OK to use ™ if you are unsure.

http://www.google.com/permissions/trademark/our-trademarks.h...

http://www.google.com/permissions/trademark/rules.html

> I don't see how he is violating the trademark

AFAIK, without a trademark license, using it to promote your product (including using it as part of your product name) is generally a violation of trademark. Nominal fair use allows you to use the trademark to refer to the thing that trademark names in descriptive text, but not generally as part of the name of your product.

Reply on news.ycombinator.com