Hacker News Reader

Comment by wrs | Hacker News Reader

The “Why?” section of the home page is rather disingenuous, since the Economist is not “SEO optimized garbage”, nor does it want you to “sign up for some newsletter”. It’s just an excellent newspaper that needs to pay the staff that writes all those articles. Just like 12ft.io needs to pay its hosting provider.

View on HN

I'd be fine with that if they didn't let search engines crawl their content. If they want to be on the web, they have to play by the rules of the web. Instead what they want is to reap the benefits of the web while refusing to participate in the open web.

What the heck are the “rules of the web”? Only valueless information is allowed to be indexed?

Are you offended that when you search for a book on Amazon you have to pay to read the whole thing?

> Are you offended that when you search for a book on Amazon you have to pay to read the whole thing?

You don’t. ;)

And, those are the rules of the web.

Don’t understand the downvotes. There are legal ways to read books (https://www.overdrive.com/apps/libby/) and the ability to straight up not read the books but read summaries instead. The internet has provided this free flow of ideas and repackaging of thoughts.

There are also ways to read books that circumvent copyright, but in the spirit of this discussion, that’s not necessary to dive into.

It's a bait and switch. They give the search engines full text to crawl, and then nothing when a user clicks that link from their search. They got the tangible monetary benefits of having a high Google search ranking, without giving access to the actual content, which is how the web is supposed to work.

It's not my fault that news websites have continually debased their users with horrible advertising and tracking, to the point where ad-supported news is pretty much no longer viable. There are plenty of examples of smaller content offerings like podcasts which do quite well being ad-supported, because they have the trust of their audience, and they only accept quality advertisers so they can charge a premium for their ad space.

Books are different, they are not the web. I find things like Google Books very helpful, even for things like when I'm reading a physical book and want to look up which page I read something, even though I can't read the full book through it.

> They got the tangible monetary benefits of having a high Google search ranking, without giving access to the actual content, which is how the web is supposed to work.

That’s an opinion. Another opinion is that the web is a hypertext system and that Ted Nelson, who defined that term, included a royalty mechanism (https://en.wikipedia.org/wiki/Project_Xanadu#Original_17_rul...)

You seem to have the misinformed belief that Google and search engines generally somehow make up the fundamental fabric of the web? Google is just a company that has a pretty good catalogue of what they've been able to piece together of what's available on the open web. Crawlers are no more a fundamental part of "the open web" than anything else.

GP's complaint is that these publishers are serving different content to Google than they serve to people, and Google knows this but pretends that people will get the indexed page. This is the search engine's fault. If it knows I will get a different result because I neither have nor want an account, it should let me hide that result, so I won't waste my time.

Chances are even if google crawled 12ft.io it wouldn't show in the results because it'd be duplicate content.

Meh, to each their own. I’m inclined to agree with 12ft.io on that front

I'd be happy to pay The Economist but I won't buy an expensive subscription because I don't read enough of their articles to make it worthwhile. I'd like a browser based wallet which I could use to pay per article with one click.

This would work with true micropayments, i.e., payout in mills, not cents. You wouldn't even notice it, but like streaming, magnified by 1000's, eventually it's real money for the content provider.

I like the model that Yalls.org implements: https://yalls.org/articles/97d67df1-d721-417d-a6c0-11d793739...

"Continue Reading: $0.06 USD"

Then search engine crawlers should get paywalled too.

The motivation is correct. You run a search on google and get mostly paywalled content. I'm fine with news sites requiring subscriptions to view their articles but they shouldn't also get the benefit of being listed at the top of search results for key terms.

Alternatively, the search list should show if content is paywalled or give you search options to remove paywalled content.

I'm not following your logic here. The Economist wants to charge readers because they produce high quality content that's worth paying for (in their estimation). This seems entirely orthogonal to whether they blacklist a web crawler - a crawler they didn't even ask for, which would be all over their website whether they want it or not.

I think you're confused because the crawler and the browser both use the same channel and the same protocols to access the information (the website over HTTP). But that's just a detail. Google could send them a hand written form for them to fill out with details of each of their articles and some thumbnail images to be manually entered into a Google database for all we care.

And surely we'd want high quality content returned from a search engine. If Google never returned results where the company wanted me to buy something it'd be pretty sparse

Seems like you should take that up with Google? Why should The Economist be obligated to serve no content to a crawler just because they want to charge readers a fair price for their content?

I think the person you're replying to is agreeing with you.

IMO the issue is the paywall is essentially "cloaking" by google webmaster standards. different content is displayed to the crawler (actual text of the article which gets indexed) vs the user (a paywall).

the content provider might not ask for the crawler but they are certainly catering to it - and benefitting from it.

> I think you're confused because the crawler and the browser both use the same channel and the same protocols to access the information (the website over HTTP). But that's just a detail. Google could send them a hand written form for them to fill out with details of each of their articles and some thumbnail images to be manually entered into a Google database for all we care.

I disagree; I care quite a bit about whether the Google results are about the actual page I'm going to see or about what the page author claimed the page would be about. (Indeed I'm old enough to remember that what originally set Google apart from competing search engines was that it would ignore the meta keyword tags that authors used to describe their pages, in favour of indexing the visible page content directly)

I’m not confused and I don’t really understand why you think I am.

I believe the content that is indexed is the content you can see. Sites used to be penalised, heavily, for returning different content to google. Hiding the paywall for google falls into that bucket.

At a minimum the search results should display if they’re paywalled and provide tools to exclude that content from results.

You're a couple lines of robots.txt from not having your site appear in search engines most people use. Meanwhile, putting something up on the open internet includes the risk that people and robots will see it.

I can see Google offering “private indexing” to big publications for a fee in the future. Keeping 12ft ladder out of the loop effectively.

1. I pay for The Economist, I want to share articles with my family & friends without them having to jump through the login hoop, until quite recently there wasn't an intention method to do this[1]

2. My partner is an annalist, her work is often quoted and republished in part or in full behind paywalls, going through correct channels to read what they published can take days to weeks & they need to know Pronto!

Edit: Clearly because I'm paying for The Economist I believe they should be remunerated for their journalism, but a hard policy of "no access unless you've verify-ably paid" would be a worse status quo

[1] https://web.archive.org/web/*/https://myaccount.economist.co...

It's disingenuous to tease content that is behind a paywall.

Someone better tell every movie studio on Earth that's it's disingenuous to make movie trailers.

Surely you don’t need someone to point out to you that those aren’t the same things.

You're right. One of them is professionally produced content, advertised with a preview to whet your appetite, where you're expected to pay for the right to consume it fully. While the other is.... uhh.... the same thing.

Guess that’s the thing. When it shows up in your search result, we do expect it to be ready to read without paying for it. This is partly Google’s fault for not making it easier to exclude paywalled content. Would paywalled content-providers like Google implementing that feature? I doubt it. But also we can use something like 12ft.io.

When you click a click to a trailer, you know it's not the whole movie.

When you click a click in Google, you don't know if it's paywalled.

How can you not see the difference? It's basic.

Frequently movie trailers are very disingenuous. They include scenes not in movies. They emphasize minor scenes cut to present a different view of the movie to different audiences, often to the point of completely misleading at least one population as to what the movie is about.

Reply on news.ycombinator.com