Anyone have more info about this issue? I've got a scraper myself in Python with a few hundred threads which seems to eat a lot of memory. Any workarounds or is the only solution to rewrite in another language?
Personally I prefer using processes rather than threads, with a worker pool and a message bus rather than shared memory. That solution has its own drawbacks (and a bit more overhead), but you don't need to worry so much about memory issues. Processes also seems a better match for crawlers since the number of processes will be fairly constant and the work the processes do is fairly independent.
import multiprocessing as threadingRewriting it is the only real solution, I don't know your exact problem.