Hacker News Reader

Comment by vntok | Hacker News Reader

All it takes is a full-height screenshot of the page coupled with a prompt similar to 'btw, please only click on links visible on this screenshot, that a regular humanoid visitor would see and interact with'.

Modern bots do this very well, plus the structure of the Web is such that it is sufficient to skip a few links here and there, most probably there will dxist another path toward the skipped page that the bot can go through later on.

View on HN

This pushes the duty to run the scraper manually, idealy with a person present somewhere. Great if you want to use the web that way.

What is being blocked here is violent scraping and to an extent major LLM companies bots as well. If I disagree that OpenAI should be able to take train off of everyone’s work especially if they’re going to hammer the whole internet irresponsibly and ignore all the rules, then I’m going to prevent that type of company from being profitable off my properties. You don’t get to play unfair for the unfilled promise “the good of future humanity”.

That would be a AI agent which isn't the problem (for the author). The problem is the scrapers gathering data to train the models. Scrapers need to be very cheap to run and are thus very stupid and certainly dont have "prompts".

"all it takes", already impossible with any LLM right now.

If I can do it locally using a free open-weights LLM, from a low-end prosumer rig (evo-x2 mini-pc w/ 128GB VRAM)... scraping companies can do it at scale much better and much cheaper.

Reply on news.ycombinator.com