Hacker News Reader

Does this handle something like Distil [0]? Or is that type of scraping not the focus of this product?

[0] https://www.distilnetworks.com/block-bot-detection/

No, it won't. However, Distil is not hard to work around if you automate a real browser in headful mode.

Could you point me to some reasonably straightforward ways to do that? Thanks!

I don't think there's any HOWTO posted online; I just worked it out by trial and error.

Use a real version of Chrome (not Chromium) and headful mode. Mask the navigator.webdriver property. Pace your requests and take care to use "good" IP addresses.

Keep in mind that as soon as Distil sees something obviously automated (like a headless browser) the source IP address is "burned" for some number of days.

Thanks!

I second this!

Reply on news.ycombinator.com