Hacker News Reader

In my experience bot detection is moving more towards looking at network activity and IP reputations. Using a proxy will go along way, it's easy to implement, and the cost can easily be passed onto the customer.

View on HN

In my experience, the thing that makes me actually have to lug out my ol' headless browser is that a lot of websites are starting to implement obfuscated cryptographic puzzles in their JS, making it really difficult to emulate without just running it in a browser.

Starting to? That's been going on for at least the last 3-4 years. Akamai tends to rely on that more than Cloudflare and in my recent experience Akamai is winning that game and browser emulation alone, headless or not, isn't going to bypass Akamai. Recently I have seen browser emulation not be effective at all for bypassing bot detection.

The only kind of emulation where I have seen success is mobile and in that case you need to run a device emulator.

In my experience it's starting to moving towards "use a headless browser with patched attributes" or nothing else will work.

edit: I have quite a bit of experience with Akamai and other vendors =)

Reply on news.ycombinator.com