Hacker News Reader

Sounds like pretty sophisticated filtering!

I’m working on a scraping project at the moment so looking at this too but from the other end. Super low volume though so pretty tame - emphasis on success rate more than throughput

I bought a 4G dongle for use as last resort if nothing else gets through. And also investigating ipv6

View on HN

Using a 4G dongle makes it easier to hide in the crowd indeed. Since your traffic will go through heavily shared mobile IPs, probably with thousands of users behind them, anti-bot vendors won't/shouldn't block per IP, but per fingerprint/session cookie instead.

Ah hadn’t realised it’s the NAT. I thought it’s because the IPs are dynamic and rotate too much. Interesting.

Currently planning on doing a layered approach. Cloud IPs first etc.

Interesting challenge but also trying to be somewhat respectful about it since nobody likes aggressive bots

Reply on news.ycombinator.com