I've had great success with Ruby/Mechanize for regular html scraping and phantomjs for dynamic page scraping.