Paraphrasing newrelic "it takes a village to count our proxy nodes".
We have our placing orders infrastructure on AWS, and whole in-house cloud dedicated to product crawling built on top of Digital Ocean.
hm, I didn't mean infrastructure, I mean, did you buy proxy nodes from someone like sslprivateproxy and slap HAProxy in front...
Most ecommerce sites wise up to bots crawling them and have a robots.txt that suggests you might not want to...