Does Upton have any way of dealing with JavaScript or ajax calls? For a lot of the scraping I do in Python, this is crucial for me. I use Selenium's Webdriver (along with beautiful soup or lxml) now for that - definitely open to other options.
While I'm sure this would be possible, I think a node.js scraper would probably be a better fit for that sort of a project.
Node, by itself, doesn't have full versions of a lot of the objects that Javascript on the pages would refer to (DOM, event model, etc.); phantomjs gives you all of that.
We maintain a Node to Phantom bridge for this: https://github.com/baudehlo/node-phantom-simple
What about using something like node-gir, or whatever appjs does to combine the event loops of node/v8 and chromium/v8?
Phantom has some quirks but overall it's pretty solid.
If you insist on controlling webkit through javascript, you can use gnome-seedjs. But this is problematic/annoying because there's no commonjs implementation yet.. in phantomjs you can require() node modules in the outside context. Not so much in gnome-seedjs..
Also, X means it's not actually headless, even if you're using xserver-xorg-video-dummy or xvfb. For this reason, phantomjs got rid of the X requirement a number of versions ago.