Hacker News Reader

Wow!

Thank you so much for this, I will for sure try and play around with this.

Is it possible to run the Crawler behind like socks5 proxy or such?

You can certainly change the crawler's database connection from "localhost" to an IP address on a different machine, but I am unsure how that works with that type of proxy (I had to look up what a socks5 is). Sounds like it can work though.

Well what I meant was, say I have a server with the ip 13.223.12.212 and I want to run the crawler there, however would like to crawl the actual websites with the ip 23.215.23.15 (aka my proxy, socks5 being one of the several protocols to do it)

If you get what i mean :P

I assume it's possible if I just change some of the curl options in the crawler code

Do let me know if you succeed

Reply on news.ycombinator.com