Phantompy - Headless WebKit engine for Python, like phantomjs | Hacker News Reader

Phantompy - Headless WebKit engine for Python, like phantomjs

github.com

171 pointsjuanriaza45 comments

I wonder if Phantompy is truly headless so it works on servers without GUI libraries and without xvfb.

QtWebKit needs access to some GUI features out of the box, but phantomjs includes platform abstraction code that avoids the dependency on an X server. That's the feature that made me migrate my own C++ crawler code to phantomjs.

Actually, you don't even need xvfb these days (and I have no idea why anyone is telling people that they do). You could use something like xserver-xorg-video-dummy. Of course, as you have pointed out, PhantomJS doesn't require an X server (as opposed to these other projects, wtf) so that it is "really really truly headless".

OT I know, I cant help but feel a little smug when I see cool projects like this and they use cmake, and not autotools.

Thank you cmake, you've made working in the C ecosystem fun again.

Check out gyp and ninja

https://code.google.com/p/gyp/wiki/GypVsCMake

https://github.com/martine/ninja

CMake is pretty horrifically inefficient at compiling WebKit. The makefiles it generates fork to call CMake (to do things like printf in color) far more than they fork to call the compiler. The ninja generator is better, but it has various issues with long command lines on WebKit currently (trunk CMake fixes some of these, but then it doesn't make some directories in the output directory).

Also, the CMakeLists syntax is pretty nasty IMO, but it feels like gyp, cmake and autotools are in some kind of syntax ugliness competition... (Maybe because nobody ever wants to work on a build system, so they'll just do this one little hack which inevitably grows tentacles).

Is there anything wrong with the python bindings to phantom? I used phantomjs (it's great)a few years ago and I thought there were some python bindings to the original phantom C++ library - could be wrong though.

Ariya Hidayat & co have put a monumental amount of effort into exposing the inner workings of webkit in phantomjs - I'm not sure I see the necessity for a python rewrite. Taking a quick look at the code, most everything I use phantomjs for outside of screenshotting is not available. A great start for sure, but unless there are some fundamental shortcomings with phantomjs that I'm unaware of, I agree with boothead that python bindings are probably the way to go.

One thing that jumps out at me is that phantompy uses Qt5 internally, while phantomjs uses Qt4. There are lots of new WebKit features and optimizations in Qt5.

phantomjs will target Qt5 in the next release, 2.0.

There were a set of Python bindings for phantom.js named PyPhantomjs, and it was included in actual phantom releases, but the maintainer moved away from the project, no-one stepped forward and it was ultimately removed from the project (https://github.com/ariya/phantomjs/issues/10344).

So this project is really useful.

> no-one stepped forward and it was ultimately removed from the project

Here's the source code, if you are interested:

https://github.com/kanzure/pyphantomjs

Nice. There's also https://github.com/jeanphix/Ghost.py which I remember using a couple of months ago.

Ghost.py seems promising but is painful to work with. Installing the dependencies took about an hour (compiling PyQt itself took about 25 minutes). When we did finally get it working, PyQt would crash after the third or fourth scrape and would take between 15 to 30 seconds per scrape. Where we used it: https://github.com/createch/PriceChecker.py/blob/master/pych....

My advice to anybody looking to do headless webkit in Python: Don't use Ghost. Try out PhantomPy and try using the Selenium webdriver for Phantom.

We switched over the Selenium webdriver for PhantomJS and found it to be much more stable and fast. See this SO answer by another Ghost.py user who gave up: http://stackoverflow.com/a/15699761/854025. Where we used the Selenium driver: https://github.com/createch/PriceChecker.py/blob/phantom/pyc....

Just to give another perspective: I'm not sure on which OS you had those issues, but I'm successfully using Ghost.py on Ubuntu and OS/X. On OS/X it was harder to install PyQT and sometimes there were weird issues, but on Ubuntu it all went smooth.

By now I've ran it on thousands of different websites with little issues. It's true that I'm "recycling" the processes after ~10 sites, but that's mainly because of memory leaks.

EDIT: and to clarify on Ubuntu installing either PyQT or PySide (Ghost.py now supports both) was as easy as apt-get install...

I tried (but failed to) get it to run at all on OS X. We were able to get it to run on Ubuntu, but thats where it crashed after every three to four scrapes.

When was it?

About three weeks ago.

There are so many half-baked headless browser projects, many building in some way on PyQT. I've made heavy use of Spynner and Ghost. Spynner especially. It always seemed to offer exactly what I needed but the inconsistent docs and endlessly-deep stack always seemed to leave successful implementation just beyond my grasp.

Now, as an OSS maintainer myself, I sympathize. But I'd love it if this or another one of these projects started breaking away from the pack and could build some critical mass. I'm all for democratization but in this case I'd gladly trade 10 mostly working projects for a single solution ran by an iron-fisted BDFL if it meant decent docs and a mostly stable API.

I've been working on a project like this that's nearing completion (it works, and the api is mostly complete). It differs from this in that it uses any normal browser to fulfill requests (via extensions).

The idea is to have a pythonic API to any browser that has reasonable support for extensions; i.e. a cross-browser scripting.

What does it offer over Selenium Webdriver? Yeah, the API is not terribly pythonic, but it's battle tested and stable, works across almost all browsers and there's always wrappers like Splinter (http://splinter.cobrateam.info/) if you prefer a more pythonic interface.

For those of us on the outside, what's the typical use cases of a headless browser engine? Testing?

And web scraping

We use it to generate images from a website. We do a product with HTML5/CSS3 and webfonts, layout it all then "screenshooting it" to make a final image with the proportions we want. It's cool because its server side and we can generate endless images based on what the user want.

Addendum: If you want to use webfonts, specially from Google, you need to recompile Phantom to support WOFF.

I was going to ask about off-screen rendering. What kind of speeds are achievable? 30fps? Could I use this to stream a web texture into a game?

It is not fast as we wished. The whole request and image deliver is about 2 seconds or more, It's because we're using right now the same server for everything but for what are you asking, I think is not good idea. Achieve 30fps i see it hard.

Can you capture PDF that way, or is it only png?

To do PDF, you better install HTML->PDF command line tool than that. What this do is simply load a webkit browser and we just perform an screen of that.

Printing works differently in different engines, so I want the designer to use her HTML / chrome skills to design templates, and then render the PDF with users data using the same engine. Command line HTML to PDF tool will work too, but it will require the designer to relearn engine quirks.

http://code.google.com/p/wkhtmltopdf/ hasn't gotten much love lately, but it could still work for you.

Running JavaScript unit tests headlessly in CI.

I use phantomjs to login to my bank's website and grab my balance for me.

Unfortunately my bank decided to outsource their web operation to the cheapest bidder from India and as a result their web application is beyond shit. It requires javascript to work, and therefore I needed a headless broswer engine to scrape it.

All the banks I have used in the last 10 years have required hardware tokens. Recommend you change banks even if that makes screen scraping your balance impossible....

Two factor auth is extremely uncommon in the US for banking.

My UK account uses 2 factor but, none of the financial sites I use in the US (4 different companies) have it.

Just to add though. Wire transfers require txt or call authentication.

angularjs and other JS rendering techniques make the plain HTML look a mess. Google will not be able to present the page or get its content properly, so you use something like this to take snapshots of your website post rendering and expose it for indexing & website previews. (https://developers.google.com/webmasters/ajax-crawling/docs/...)

I would be surprised if Google was not doing this themselves to some extent.

They are not (although they definitely should be). Google will read and index javascript, they do not execute it.

There's some evidence that they are.

[1] http://searchengineland.com/google-can-now-execute-ajax-java... [2] http://www.seomoz.org/ugc/can-google-really-access-content-i...

I recently built a service using PhantomJS for a client. They have graphs and charts generated with d3.js that they wanted to add to their user-specific update emails.

I built a backend service using Akka[1] and PhantomJS which would grab the appropriate user's page, render it to HTML, splice up the images and save everything to temp files. Then I had a Java process that read those back in then format and sent an email to the user.

[1]: http://akka.io

OT - I've been looking for a headless WebKit for Go (Golang) I've searched all over with no luck. If anyone wants to become famous, please make one :)

Does this actually offer full DOM access?

I see a 'cssselect' in tests.py, but I wonder how much of the DOM Api is actually supported?

Take a look at lib/webelement.hpp -- there's not too much yet, notably absent is a way to iterate children. But you can evaluate Javascript.

Cool you can take screenshots with it, see the test_capture_page example.

Will this render Flash content?

Looks phantastic

Reply on news.ycombinator.com