Bypass Paywalls – A Firefox extension to bypass paywalls of many news sites | Hacker News Reader

Bypass Paywalls – A Firefox extension to bypass paywalls of many news sites

addons.mozilla.org

74 pointsknown63 comments

Correct me if I'm wrong but I think publishers are trying to have the cake and eat it too. They want their content to be discoverable in search engines. They also want readers to pay for it. Given how search engines currently work this is a self-contradictory model.

I imagine search engine could offer publishers an API to send content for indexing without having to publish it. Google and others could even charge publishers for that and then published could ask readers for subscriptions.

I absolutely support such programs. Similarly to adblockers and DRM, they show that the mechanisms used to generate money simply does not work. Technically it is impossible to expose content to a (web crawler) robot but not to a human (inverse CAPTCHA). And technically it is impossible to control pixels on a free user device.

Publishers, if you want to have people pay for your content, make honest paid subscriptions and deal with it that you vanish from the openly accessible web.

Not necessarily. I could imagine Google adding a "paid crawling" service where you tell Google how much your content costs, explicitly allow them access via some authenticated method, and then they display the price next to your content in search results.

You're imagining that it can't work because it can't work in a generic way with all search engines, including future ones that don't exist yet. But that doesn't have to be the case. Most content providers only care about Google.

It’s hardly having their cake and eating it too. It’s the equivalent to wanting placement at a news vendor. Your API idea isn’t a bad one, but I hardly think that these news organizations are at fault for wanting to be discoverable, while also wanting money in exchange for their hard work.

What I find most interesting is the very clear but often ignored 'Cloaking Guidelines' by google:

"Cloaking refers to the practice of presenting different content or URLs to human users and search engines. Cloaking is considered a violation of Google’s Webmaster Guidelines because it provides our users with different results than they expected.

Some examples of cloaking include:

...

- Inserting text or keywords into a page only when the User-agent requesting the page is a search engine, not a human visitor

" [0]

Google is happily showing LinkedIn, FB, pinterest and news sites content. But when I, Joe User, go to the page, I see nothing but some login/register/pay now form. How is this not a violation of the cloaking guidelines? Clearly google is getting different content than what I am!

(Presumably this is how article's extension works... by masquerading as GoogleBot -- again proving that these sites are serving up different content)

[0] https://support.google.com/webmasters/answer/66355?hl=en

edit: formatting

I find it much easier to tell firefox to block cookies and localstorage for such websites. Usually works when they have a certain free allowance. You can do it from the address bar quite conveniently.

Self-destructing cookies (whitelist only) + firefox containers works well for me

Or we could pay for journalism that the world badly needs.

Ah but that doesn't scale!

Do you have an example?

Blendle (https://blendle.com) is a news aggregator where you pay per article that you decide to read. Price ranges from ~20 cents for a fluff opinion piece to ~90 cents for quality journalism. It’s just for the Netherlands and possibly Germany (?) atm but they are working towards a US launch.

This is a pretty good model I think. It’s accessible and sustainable for both publishers and readers.

My problem with Blendle is that you're essentially paying for them to get a full profile of every article your read. At least Google Analytics and such can be blocked, making it more difficult to connect your profile across the sites of different publications. With Blendle, that's impossible.

I know it's adding complexity and most people don't care, but I wish they used a system of crypto vouchers (not like cryptocurrencies, more like Mozilla Persona, which allowed identity providers - in this case, Blendle - to vouch for the user without knowing to whom they were vouching).

Brave browser has a micropayment system. I would like to see this succeed. I'm happy to pay a little cash for content, just not in the form of malware and irritation.

https://brave.com/publishers

I love brave.

Is there an ethical journalistic entity that does not run on ad revenue, but only paid for by readers?

Go to the shop, buy a newspaper.

They still sell ads, they still do bad journalism with clickbait etc.

Why doesn’t it scale? People used to have to pay for access to all of these articles (and still do BTW). It’s actually become way cheaper to read a wide variety of sources.

Your statement comes off as really entitled.

I don't consume things that I'm required to pay for and then don't pay for. So no on the entitled issue.

Paid journalism (by the consumer) is getting harder and harder. Do you have a good example of journalism today that is paid for by the consumer (i.e. monetarily) that is good journalism?

(And I consume a wide variety of sources to specifically try to pick up on bias, and oh boy!)

WSJ, The Economist, Harpers, Pro Publica, etc.

I'd rather not pay for it with my privacy though.

I agree.

I developed a system for subscription payments that (1) is resistant to falsifying visits and (2) preserves user privacy. I got a US patent late last year[0], so that I might enjoy being a bit of a lightning rod for downvotes.

The gist is that the publisher can ask "is this person a member of the subscription pool?" and get a yes or no. And that's all they can get. I can't easily prevent them from using other mechanisms but I can refuse to make the problem worse.

[0] https://patents.google.com/patent/US9853964B2/en

That seems interesting. Are you working on it still?

On hold for the moment because of the practicalities of being a non-US citizen on a work visa. If someone ambushes me with a hefty cheque, sure, but for the moment I am limited in what I can do.

The only thing you will get from a patent on that is that nobody will use it b/c it's patented. So you're effectively blocking one road to solving the problem. There are others, of course - it's not like there's just one way of doing that.

This is a double edge sword. We complain about too many adverts on sites, and we complain about paywalls. Without ads or paywalls, how do big sites pay for themselves? If you want quality reporting, then someone has to pay those reporters a fair wage for their skill set.

As much as I'd like 'information to be free' we do not yet live in a 'Star Trek' world where money no longer has any meaning, and where people produce 'stuff' purely for the benefit of themselves and others.

Until we reach that utopia (or dystopia depending on your view) you can't have your cake and eat it; It's Ads or Paywalls, pick one.

When I hit a paywall, I walk away, or pay up, because that's the ethical thing to do - I do expect no tracking though, when I do pay up. Reasonable ads (fast, 1st party served) would be ok, tracking is really not.

But... bots (google, etc) are allowed to get through. I understand the reasons, but then, is it really a paywall? To be honest, I won't bother with cookies and other tricks: logged in, allowed, logged out, walk away.

In the end, I agree: don't bypass paywalls, the whole adblocking argument becomes invalid, if we do. Some publishers are trying to find a new way of income, and bypassing that invalidates the efforts.

(EDIT: treating bots differently is wrong on so many levels. I changed the User Agent of my miniflux RSS reader to "miniflux-legacy (Googlebot for Tumblr)" from "Miniflux (https://miniflux.net)" because otherwise Tumblr shows GDPR cookie consent interstitial for RSS feeds. There really, really shouldn't be a separate version for bots.)

I agree completely in that high quality journalism requires the appropriate wage, and I detest advertising. However, playing devil's advocate, the flip side of pay walls is that you essentially remove/hide quality journalism from being used in the the argument against the poor quality, free to publish 'news' that is often referred to as 'fake news'. Just a thought.

Paywalls that actually work.

The existing model is to treat the Internet as a magazine rack. It doesn't work that way.

A user might read one article from the WSJ a month. Obviously they won't pay 5 USD for that, they'll either attempt to get around the paywall or not bother.

Giving users a few free articles means that they'll just rotate around sites to get what they want. You need to charge a small amount from hit 1.

Not withstanding that it's not hard to build a paywall that actually functions, just don't send the content unless you've paid. This addon relies entirely on the fact that content which has not been paid for gets sent anyway.

Realistically though, the answer is that paid journalism disappears, or the Internet as we know it disappears. Increasingly lately it's looking like both will happen.

> Realistically though, the answer is that paid journalism disappears, or the Internet as we know it disappears. Increasingly lately it's looking like both will happen.

Seems a touch hyperbolic. My question would be: what part of the internet depends on paid journalism and how would the disappearance of paid journalism effect e.g. buy stuff online, looking at pornography, or accessing social media?

Please re-read my comment. I have not stated that they are dependent.

It's Ads or Paywalls, but it's not Paywalls or Bypass, you can choose both. Since micropayments are not here yet, and most of us can't afford subscriptions to every single site, an approach is to just choose a few you can afford, and bypass the paywalls of the rest.

I know this is unacceptable to those who have a moral opposition to accessing content without accepting the rules, but to those who are just concerned about funding journalism, this approach is essentially equivalent to not bypassing paywalls.

Also, there's a problem with paywalls, which is the enhanced tracking (by subscribing, you're giving them an excellent profile to attach to their analytics). Some people might therefore prefer to use a Bypasser even if they are subscribed.

I'd sooner just not use those sites

I'd rather just have access to the information I want.

By any means necessary!

That's implying you somehow have a right to that content - it's a similar argument that e.g. movie and music pirates use, but which they're not aware of themselves. It's feeling entitled to the content, and/or fear of missing out.

But news flash: you are not entitled to something if you don't want to pay for it.

You're not entitled to decide what I'm entitled to.

Not as an individual no. But maybe you want to replace "entitled" with "freedom" if you're arguing from a libertarian perspective.

I guess I just don't care?

I don't understand the point of bypassing, the creators need to make money, if you can afford it, subscribe, if not, use another website. it's that simple

I subscribe to a couple of sites. I won't be subscribing for an extra monthly fee I can't afford just to read one or two articles linked on HN. Therefore I have two options: don't read the article, or bypass the paywall.

In both options, the outcome for the site is exactly the same. So, not being a masochist, why shouldn't I choose the outcome that benefits me without harming anyone else?

But it is harming someone else in a way, they only want you to read it if you pay for it, there will still be bandwidth costs to cover for people who bypass the paywall, some might argue that in some paywalls, the content is already sent to your browser anyway but that's a whole different story.

It's the same argument for piracy, "if I make a copy, I'm not taking away the original, so I'm not stealing", which is just a lie to tell ourselves to feel a bit better at the end of the day.

I agree it's the same as in piracy, and I do the exact same there - pay for what I can, download the rest.

Regarding the bandwidth costs, those are absolutely negligible, since the methods for bypassing the paywalls either only fetch the text (e.g. http://outline.com) or cache it on other servers (e.g. http://archive.is).

It is definitively not what they wish, but not complying with someone's wishes is not harming them.

On that last point, while I don't take my moral code from the law, I note that it agrees with me; for example, even if you have a contract with a guy to build your house, and you say your want a certain brand of pipes, but he ends up using another, you're not entitled to anything. Unfulfilled wishes are not harms.

Technically, the extension is clearing cookies for the site, and setting the "Referer" header to either google.com or facebook.com. Is it illegal to do that?

Source: https://github.com/iamadamdev/bypass-paywalls-firefox/blob/m...

Cookie clearing is almost certainly not a legal issue; there'd have to be an explicit legal agreement between the setter and the keeper to be an issue in the first place.

The "Referer" header is an interesting question. Hypothetically, if there's be some kind of renumeration for the site (eg: "I grant access to all of your users to my site and in exchange, for every referred visitor, you pay me 0.01 cent"), then spoofing the Referer would incur a financial cost to Google/Facebook. The incurred cost would be negligible for the individual user, but a case could be made against the extension publisher for the aggregated costs.

Illegal?

What?

Wallace and Gromit in a case of... the Wrong Bits!?

IANAL but the definition of Wire Fraud is ridiculously flexible. Manipulation of http headers to get free access to paid services is likely to get you convicted if anyone cared enough to prosecute...

Also likely to be punishable under DMCA as circumventing access controls.

Blocking ads being about privacy always seemed like hypocrisy and extensions like these only prove it.

Of course, the ads revenue model for the Internet only happened because people basically want free labor, being unwilling to pay for the content they consume, screw the publishers the world will survive.

It's sad because what we'll get is legislature and/or more DRM.

This extension proves nothing; it's perfectly sensible that someone worried about the privacy of their reading doesn't want to make it worse by tying a payment ID to their profile. Subscriptions are in that sense worse than ads.

Haha, yes, exactly my point.

And this makes me think that the existence of DRM is justified.

How does this work? Do they reuse login credentials or something like that?

Probably using a trick to go there via Facebook, there was a trick like that somewhere. Or imitate a search engine, because the news outlet also wants a high Google ranking. Or use a copy from archive.is.

Depends on the website usually, one that I managed to "hack" is a newspaper in my country (the biggest one actually) that only hides the article text through a CSS class; You just remove that from the HTML through dev tools and have access to the whole thing. I hope in most cases it is more complicated than that

I want quality journalism but exporsure should be payment enough.

Exposure doesn't pay many bills.

There's an /s in there right?

In which jurisdictions is this deliberate circumvention of access controls legal?

The effectiveness of the access control is usually a factor.

If there is just a banner hovering over the actual text, and the extension merely removes that banner, than one could question whether there even was an access control in the first place.

As an extreme example, a Finnish court ruled that CSS (as used by DVDs a long time ago) was ineffective.

https://www.turre.com/finnish-court-rules-css-protection-use...

I agree that a banner or any over overlay is not protection - the content has already been delivered to the recipient.

In Germany, this is defined in § 95a UrhG [0], as in "bypassing safeguard measures to gain access to copyright material". "Anti-Anti-Adblocks", as in Adblock filters bypassing adblock popups, were already declared illegal in the BILD case [1].

[0] https://dejure.org/gesetze/UrhG/95a.html

[1] https://www.wbs-law.de/it-recht/verbreitung-einer-anleitung-...

It still depends on whether the protection is considered effective.

In the case of this addon, the paywalls are often just overlays that you can also remove manually with a few clicks.

Whether or not clearing cookies would count as bypassing is an interesting question, because Adblockers themselves are legal. The specific BILD case is about blocking and disabling a tool to detect adblockers, in which case it fully denies access to the article.

Ie, the anti-anti-adblock is actively interfering with the site's function on the client side.

Cookies and referer are handled on the server-side and outside the user's control.

I would question if the last sentences of 95a apply to setting a referer and clearing cookies. They are more closer to tampering with computer system (263a).

The first word of the law is "wirksame", which means "effective".

It's very easy to argue that this protection that is trivial to circumvent is not "wirksam".

Yes, that's why I've added the second link. Someone posted a tutorial on how to bypass the BILD Anti-Adblock. He argued that this is not an "effective" protection, but a court ruled an "Anti-Adblock" script as "effective".

It depends on the court. IIRC, that ruling came from the Landgericht Hamburg, which has become a bit of a running gag because of their copyright friendly rulings.

Reply on news.ycombinator.com