Hacker News Reader

You probably want the web equivalent of malicious compliance - an algorithmically generated web-hole or similar. That way the bot author isn't entirely sure you're on to them; it could be a bot or server error. Like send the right headers but garbage data that looks like it's compressed but isn't, or doubly compressed garbage, or trim pages at a different place (before anything interesting), or slow data transfers, or ...

View on HN

The main thing headless chrome saves is having to spawn and manage xvfb per instance.

If more sites adopt serving fake data to headless chrome, people will just return to the old xvfb workflow for those sites, and use headless for everything else.

Could even use a small set of xvfb scrapers to verify the results, and automate detection of false data.

Just randomize the content they're trying the scrape imho

I saw that with a page and it works extremely well. Inserting wrong data based on rules (there it was on rate limiting rather than user agent) is extremely hard to detect. The scraper never knows if they see real or wrong data.

On the other hand, this will also get wrong data in search engines.

I think you could get pretty far just by simply scrambling the CSS classes and ids.

Reply on news.ycombinator.com