Hacker News Reader

Comment by jpablo | Hacker News Reader

They aren't blocking anything. They are just asking nicely not to be crawled. Given that AI companies haven't cared a single bit about ripping of other's peoples data I don't see why they would care now.

View on HN

A number of sites have started outright blocking any traffic that looks remotely suspicious. This has made browsing with a vpn a bit of a pain.

This has been ever increasing for years now. Bots, attacks, scrapers, AI, all these things seem to be the majority of traffic on most sites.

I wish I could go back to the days of doing almost anything at all without having to tell a server what a motorbike or traffic light is.

LPT: switch to the audio captcha. Yes, it takes a bit longer than if you did one grid captcha perfectly, but I never have to sit there and wonder if a square really has a crosswalk or not, and I never wind up doing more than one.

In their attempt to block OpenAI, they block me. Many sites that were accessible just 2 years ago, require login/captchas/rectal exam now just to read the content.

Im looking forward to the life experience that is content I want to read badly enough to endure a rectal exam.

It's not that bad ...

Not sure why you're being downvoted. Watching str8 bois react with shock and horror at the idea of anything near their butt is hilarious.

Prostate and rectal cancer is real, boys. Grow tf up about it.

> captchas

I suspect that AIs are already more effective than humans at passing captchas.

That would be an example of AI providing real value that I would pay for.

These exist for a fee if you want to use them

I used 2captcha, for a fee ... it doesn't work

They block plenty and they do it crudely. I get suspicious traffic bans from reddit all the time. Trivial enough to route around by switching user agent however. Which goes to show any crawling bot writer worth their salt already routes around reddit and most other sites bs by now. I’m just the one getting the occasional headache because I use firefox and block ads and site tracking I guess.

Wouldn't it be somewhat trivial to set up honeypots?

Yeah, probably right. If you want a great rabbit hole, look up "Common Crawl" and see how a great academic project was absolutely hijacked for pennies on the dollar to grab training data - the foundation for every LLM out there right now.

It's hard to envision a greater success for the "great academic project" than what happened. I mean, what else were they trying to accomplish?

It was meant to be an open-source compilation of the crawled internet so that research could be done on web search given how opaque Google's process is. It was NOT meant to be a cheap source of data for for-profit LLM's to train on.

*edit: added "for-profit"

(Shrug) Multiple not-for-profit LLMs have trained on it as well.

If something I worked on turned out to play a significant part in something that turned out to be that big a deal, I'd be OK with it. And nobody's stopping people from doing web-search studies with it, to this day.

Reply on news.ycombinator.com