I have done some webscraping with twisted and BeautifulSoup. But it was just for a few small sites.

That was when I found http://scrapy.org/ , a complete python framework for writing webcrawlers. It looks promissing, but I haven't used it so far.