Hacker News Reader

Comment by dclaw | Hacker News Reader

This is really unacceptable folks. There are those of us that have to keep these sites up, and it's seriously been a few years of nightmare scrapers and botnets, and stupid things like this that you are trying to legitimize that will make this worse. If a site doesn't want you, you should go away. There's a reason for it. Not every website is backed by a billion/trillion dollar company with the resources to absorb things.

View on HN

Making a network request from a script is not abuse, consuming excess bandwidth is. Scrapers already spoof browser reputation and cycle IPs while abusing bandwidth. Recently I wanted to convert the results of an Autotrader filter into a table so I could supplement it with data they don’t track (towing capacity). It was two pages of results, but requesting it from a script was aggressively blocked by browser fingerprinting. I had to port it to JS and run it in my browser console manually to get the data out, wasting my time.

That’s the point. For every legitimate though ultimately unimportant use case like yours there are a thousand others who would scrape the results 86,400 times a day to either try to sell or mirror and plaster with their own ads.

It's not just excess bandwidth. It's cpu cycles, poor site coding, database issues. Like I said, not everything is backed by a huge company that has the money/hardware/engineering resources to absorb the load.

Why did you need it from a script? Right click -> Save Page As... is still a thing and works quite well in my experience, even with complex React SPAs.

CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.

The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.

Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.

It’s not that dystopian if you are given the choice to trade your attestation for more valuable service. We have been paying for the web by letting advertisers correlate our behaviors together for a couple decades now.

The concern is that attestation turns into another mechanism to monitor and control people; see how age verification turns into surveillance.

And those bot protection mechanisms and ad-enforcement layers that terrorize humans are okay or what? Yes, I accept that many pages don't want me there and just don't use them anymore, but it sucks.

I am not saying that forks like this are a good idea, but I have enough frustration with said techniques that I sympathize with the effort.

Sites that are backed by a billion dollar company typically have better antibot protection, and lot of expertise in this.

Also, patched browsers have existed since long ago, although they were not open-source.

Reply on news.ycombinator.com