great site , what im missing on the web is information
on how to build or use proxy server for use with the wev scrapers
I've always had the luxury of paid quality proxies in my web-scrapers however for article purposes I'd like to have an example of cheap or free proxies for casual usage/education. Are you using something in particular?
One idea I've explored was using VPN service as a proxy but seems like most big VPN providers are not providing proxies anymore.
Anyway, I can PM you once I figure out how to put this together properly! :)
I configured an Alpine Linux docker container with openvpn and a proxy server (I think I settled on squid for stability), and a bash script to to start up the openvpn connection and proxy server with config for both passed into the container. Then just generated a long, line by line list of every possible vpn connection config line by line, shuffled and duplicated.
Then in my outer scraping function: grab a line from the config file, start up a vpn-proxy container (passing in the config), do one page download through the container proxy and then stop and delete the docker container. This allowed me to download pages in parallel, with all connections originating from different IP addresses (as long as I made sure not to exceed VPN simultaneous connection limits).
Kinda messy, and I spent ages fiddling to get the container config just right, but it worked.
This does seem like a very big overhead just for one request: build up/tear down would be quite expensive.
I actually noticed that there are personal VPNs that do not have a concurrent devices limit. I guess your hack could be modified to startup persistent images for every VPN server and have them run forever as proxy servers! I'll tinker around with this more but this would definitely make it easier for beginners to onboard on proxy based scraping as many people have VPNs ready for netflix and such already.