Hacker News Reader

Comment by lutusp | Hacker News Reader

Nice effort, but after one successful NYT session, it fails and treats the access as though it were an end user. But don't take my word for it : try it. One access, succeeds. Two or more ... fails.

The reason is the staff at the NYT appear to be very well versed in the technical tricks people use to gain access.

View on HN

They probably asynchronously verify that the IP address actually belongs to googlebot, then ban the IP when it fails.

Synchronously verifying it, would probably be too slow.

You can verify googlebot authenticity by doing a reverse dns lookup, then checking that reverse dns name resolves correctly to the expected IP address[0].

[0]: https://developers.google.com/search/docs/crawling-indexing/...

There are easily installable databases of IP block info, super easy to do it synchronously, especially if it’s stored in memory. I run a small group of servers that each have to do it thousands of times per second.

> Synchronously verifying it, would probably be too slow.

Why would it be slow? There is a JSON documenbt that lists all IP ranges on the same page you linked to:

https://developers.google.com/static/search/apis/ipranges/go...

Sure, that's one option, and I don't have any insight into what nyt actually does with regards to it's handling of googlebot traffic.

But if I were implementing filtering, I might prefer a solution that doesn't require keeping a whitelist up to date.

The whitelist can be updated asynchronously

Maybe we could use GCP infra and the trick will work better ?

Which leads to the possibility of triggering a self-inflicted DoS. I am behind a CGNAT right now. You reckon that if I set myself to Googlebot and loaded NYT, they'd ban the entire o2 mobile network in Germany? (or possibly shared infrastructure with Deutsche Telekom - not sure)

Not to mention the possibility of just filling up the banned IP table.

Hypothetically if they were doing that, they’d only be ‘banning’ that mobile network in the ‘paywall-relaxing-for-Googlebot’ code - not banning the IP traffic or serving a 403 or anything. They ordinarily throw paywalls at those users anyway.

Some sites do work, but others such as WSJ just give a blank page. Worse, Economist actively blocks this through Cloudflare.

We need a P2P swarm for content. Just like Bittorrent back in the day. Pop in your favorite news article (or paraphrase it yourself), and everyone gets it.

With recommender systems, attention graph modeling, etc. it'd probably be a perfect information ingestion and curation engine. And nobody else could modify the algorithm on my behalf.

I get the feeling there is some sort of vector of attack behind this idea, but I'm not well versed enough to figure it out.

I think you have something here.

I don’t know about paraphrased versions but it would need to handle content revisions by the publisher somehow.

"The reason is the staff at the NYT appear to be very well versed in the technical tricks people use to gain access."

It appears anyone can read any new NYT article in the Internet Archive. I use a text-only browser. I am not using Javascript. I do not send a User-Agent header. Don't take my word for it. Here is an example:

https://web.archive.org/web/20240603124838if_/https://www.ny...

If I am not mistaken the NYT recently had their entire private Github repository, a very large one, made public without their consent. This despite the staff at the NYT being "well-versed" in whatever it is the HN commenter thinks they are well versed in.

https://securityboulevard.com/2024/08/the-secrets-of-the-new...

Because I had to learn more, sounds like a pretty bad breach. But I’m still pretty impressed by NYTs technical staff for the most part for the things they do accomplish, like the interactive web design of some very complicated data visualizations.

Wasn't the creator/maintainer of svelte on the NYT data visualisation/web technology team?

Rich Harris, creator of Svelte, worked at the Guardian. Svelte has been adopted at NYT however [0].

You might be thinking of Mike Bostock, the creator of D3.js and ObservableHQ.com, who led data visualization work at NYT for several years [1]. I'm not sure if they have people of that magnitude working for them now.

[0] https://en.wikipedia.org/wiki/Svelte

[1] https://en.wikipedia.org/wiki/Mike_Bostock

Rich did work at the NYT. I thought there was some Mandela effect going on for a second, because you misled me into believing you had actually googled it by providing sources.

Might want to update the Wiki article on Svelte, which strongly implies Rich worked at The Guardian, not NYT. The only source I could quickly find that seems to corroborate what you're saying is a LinkedIn page, but because of its bullshit paywall, there's context missing.

Yeah, my bad. I shouldn't have relied solely on the Wikipedia article and my (sketchy) memory. Rich Harris is still listed as a graphics editor on the investigative team at NYT: https://www.nytimes.com/by/rich-harris

You may be thinking of https://bost.ocks.org/mike/ who worked there for a while, D3 and Observable, among many other things.

> I use a text-only browser.

Which browser is this?

-4

You could have just said the name in less words facepalm

Perhaps they're a regular contributor to somewhere that discusses 'the browser they use' and don't want any association with it via HN.

It seems parent is more interested in riddling than informing. Their browser is likely https://en.m.wikipedia.org/wiki/Links_(web_browser)

When it comes to fruitful discussions that leaves one with satisfaction and contentment, this ain't it. This is the polar opposite. Cheers

I would hope that they’re in IA on purpose. It would be exceptionally lame if NYT didn’t let their articles be archived by the Internet’s best and most definitive archive. It would be scary to me if they had only been allowing IA archiving because they were too stupid to know about it.

FWIW, if you happen to be based in the U.S., you might find that your local public library provides 3-day NYT subscriptions free of charge, which whilst annoying is probably easier than fighting the paywall. Of course this only applies to the NYT.

In the Netherlands the library provides free access to thousands of newspapers for 5 days after visiting, including The Economist and WSJ, which actually have paywalls that aren't trivial to bypass.

https://www.pressreader.com/

I just checked and the Berlin public library offers press reader as well. Will need to check that out.

Thanks for the tip!

How do you bypass the paywall with it? I can only read the “newspaper version” on their site it seems.

Edit: Just read your comment again. I assume that’s exactly what you meant.

PressReader allows reading various newspapers (in their newspaper form), the short sub to NYT I mentioned is a bit different and gives you access to the online version.

Example for LA libraries: https://www.lapl.org/new-york-times-digital

Your local library might have a similar offering specifically for the NYT.

i tried to access my own website and it says internal server error. i also tried to access Youtube and it said the same.

Reply on news.ycombinator.com