I think the good people at Scrapy [1] have done a great job in keeping pace with making open-source crawler components that handle javascript. It is called Splash [2], and I use it for rendering out screenshots.

[1] - https://scrapy.org

[2] - https://github.com/scrapy-plugins/scrapy-splash