Thanks. I used beautiful soup for the parser, and actually have written a crawler framework for my scenario. But I was wondering whether there is any tool that could automate the selection of css selector or xpath.

I wrote a couple a few scrapers and found scrapy to be my best option