Hacker News Reader

Judging by the amount of websites that construct themselves from scratch when you open them, rather than just, idk, having some html and css, you wont get much content if you dont execute js

Even before anti-scraping, lots of SPA sites won't give you any content if you don't run Javascript.

This is about the scraping tools that are used to train LLMs, not the scrapers used by live LLMs.

Even the ones used live, why couldn't/wouldn't they run some type of headless browser with Javascript enabled, rather than just making an HTTP-request and looking at the HTML code?

That's the point, the scraping that happens live can run through a full browser, performance is not an important issue there

The scraping that's taking place to train the models is in a completely different scale and (probably) cannot really bear the added cost of PoW on each page they scrape. That's the scraping that is targeted by these tools.

You're missing the point. The approach is block anything that doesn't execute the javascript. The javascript generates a key that allows your queries to work. No javascript => no key => no content for you to scrape.

Its a nuclear option. Nobody wins. The site operator can adjust the difficulty as needed to keep the abusive scrapers at bay.