For anyone interested in a Node.js crawler, my company recently released roach: https://github.com/PetroFeed/roach