A list of headless web browsers | Hacker News Reader

A list of headless web browsers

github.com

213 pointsfanf256 comments

Shouldn't Chromium Embedded Framework be on this list? [1] That seems to be robust, well maintained and widely used by many projects [3], uses a recent version of Chrome, and it supports off-screen (non-GPU-accelerated) rendering. [3]

I haven't used it myself, but I'm considering it, so I'd appreciate hearing other people's thoughts about it.

[1] https://bitbucket.org/chromiumembedded/cef

[2] https://en.wikipedia.org/wiki/Chromium_Embedded_Framework#Ap...

[3] https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage...

It's a strange omission given that CEF is free and seemingly in rude health while Awesomium, number 1 on the list (if only by dint of its name), looks dead, even though you have to pay for it.

I'm using it for off-screen rendering. It's pretty straightforward, once past the inevitable initial hump. The code is sensible (if a little smart pointer-happy), and you're provided with a good range of hooks. Documentation is OK. Nothing you can't figure out if you've got a debugger...

(I used Visual Studio with the prebuilt Spotify builds. This addin really helps: https://visualstudiogallery.msdn.microsoft.com/a1141bff-463f...)

Especially surprising since CEF's Wikipedia page lists 73 projects that use it (many of the well known and widely used, like Spotify, Second Life, Steam Client, Adobe Creative Cloud, Dreamweaver CC, Edge Animate and Edge Reflow).

I first looked at Awesomium because it had a Unity3D interface [1], but that's gone stale (Awesomium doesn't support 64 bits, much has changed in Unity since then, not to mention Chrome itself), and some developers on the Unity discussion groups were suggesting CEF as an alternative. [2] [3]

I'm interested in integrating the latest CEF with the latest Unity3D on Mac and Windows desktop, as well as for consoles.

There are (non-Unity-specific, i.e. maybe too complex and modern) C# wrappers for CEF [4], which looks pretty elaborate, but I think it would be tighter to integrate the CEF C++ code directly as a native Unity extension, without thunking through too much Windows-oriented C# glue that might not be compatible with Unity's older version of C#.

On iOS I'm simply using WkWebKit, and WebKit on Chrome, which both support off-screen rendering.

Here's some interesting stuff comparing Awesomium and CEF and other stuff from Stratoscope. [5] It's from a 446 day old thread about CefSharp [6] so it might predate the Awesomiumocalypse. The thread is interesting reading! Really makes me wonder why Awesomium is on the list at all.

It looks like Coherent UI is a commercially supported alternative to Awesomium, but I haven't looked into it yet. Their employee said good things about CEF and offered some fair criticism:

"Hi, Stoyan from Coherent Labs here. My impressions of CEF are very good. They manage to keep it very close to the chromium HEAD, which is great. The only technical downside of CEF is the imperfect GPU acceleration. In many cases however the architecture of chromium itself is a hindrance. Their processes hog memory like crazy and introduce perceivable lag. We had to write a complete new product to solve the issue for games." [7]

[1] http://labs.awesomium.com/unity3d-integration-tutorial-part-...

[2] http://answers.unity3d.com/questions/945207/unity3d-v5-aweso...

[3] http://gamedev.stackexchange.com/questions/111846/chromium-e...

[4] https://github.com/cefsharp/CefSharp

[5] https://news.ycombinator.com/item?id=10089645

[6] https://news.ycombinator.com/item?id=10087013

[7] https://news.ycombinator.com/item?id=10096616

I made a PR [1] based on your endorsement. EDIT: it's merged. Yay, we did it HN!

[1] https://github.com/dhamaniasad/HeadlessBrowsers/pull/16

You can add "open source contributions" to your CV now ;)

I'm going to add "thought leader".

It's obviously not a headless browser but xvfb can be very useful depending on what exactly one is trying to test: https://en.wikipedia.org/wiki/Xvfb

This is the way I am going.

I spent a long time trying to find an up to date javascript engine in a headless browser a while back before I gave up and just wrote a chrome extension, installed a copy of chromium, and ran the whole thing in xvfb with the extension added by command line.

In the end unless you are severely resource constrained,it is a very good option.

I've done a similar thing. There are annoyances with both XVFB and Firefox instances clobbering each other though. For XVFB, each instance needs a globally unique display number. For Firefox, you need the "-new-instance" or "-no-remote" option, along with a unique profile (e.g. in a temp dir).

Could you say more about what you mean by "clobbering each other" here?

I run firefox/chrome under Xvfb and very frequently get unexplaine `Unable to connect to host 127.0.0.1 on port 7055` from selenium, zombie firefox and chrome processes, and hundreds of TCP ports in status TIME_WAIT.

I've always assumed it had something to do with Xvfb/browser interaction.

Firefox shares sessions (cookies, etc.) between all running instances by default.

When you run "firefox", it will check if the current user already has an instance running; even if it's on a different display.

To prevent that you need to run the firefox command with "-new-instance" or "-no-remote" (which implies "-new-instance").

Firefox also 'locks' profiles which are in use, so you need to use separate profiles for each Firefox instance which is running. You can do that by making a temporary directory "T" and passing it to the firefox command as "-profile T".

As an example, the Firefox command I'm using in one of my scripts is currently:

    timeout 60 firefox -safe-mode         \
                       -profile "$FF_DIR" \
                       -no-remote         \
                       -new-instance      \
                       "$URL"

XFVB is an X server, which programs connect to based on the "DISPLAY" environment variable. You can see what that's set to by running:

    echo "$DISPLAY"

You can set it, e.g. to ":99", by doing:

    export DISPLAY=:99

Each running X server needs a different display number; this includes regular desktop sessions (usually running on Xorg with display :0), XVFB, Xnest, Xephyr, etc.

I use a variant of the script at http://stackoverflow.com/a/30336424/884682 to choose an unused display number for each XVFB invocation.

NOTE: This is based on simple automation tasks with bash and xdotool. I don't use Selenium, so I don't know if already does any of this. Hope it helps anyway :)

+1 you can run things in real browsers. fwiw Travis CI supports it.

You can also go with a regular browser tied on a vncserver display (pros: you have real captures/video of the tests, cons: the solution gets a bit heavier than when using xvfb)

It's also fairly easy to take a capture of an xvfb or an xdummy display. Dunno about video though.

Xdummy is another headless X display that has a more modern, more complete stack.

https://xpra.org/trac/wiki/Xdummy

TIL. I did not know Xdummy. I have just followed your link. It is trivial to launch. It avoids installation of vnc, has a better compatibility and is already present everywhere.

Just to throw in another option, there's selenium grid docker containers: https://github.com/SeleniumHQ/docker-selenium. The containers are a bit heavy IMHO (I suppose because they're aimed more at 'testing' instead of scraping), but I think there are a few Alpine Linux versions floating around (and easy enough to set up yourself).

Handy for more tricky sorts of web scraping, or if you want to use chromium in mobile emulation mode (causing websites with an 'app version' to switch over to a HTTP-JSON SPA in many cases). I was particularly excited to discover you can set the 'http_proxy' env variable on a "per-browser node" basis (and do fun stuff like installing DNS caching daemons, round-robinning through OpenVPN/squid containers etc.).

I really wish we had a good set of scriptable browser engines (i.e. everything but the graphics part) for all browsers. That would make testing so much easier.

Even simple tasks like taking a screenshot of a given web page involves PhantomJS, which seems really heavy-handed.

Thanks for compiling this list but the issue I'm having with this is the lack of a rating system based on usage, Github stars, downloads, repo activity, project creation, corporate support, etc. to get a measure of how reliable those tools are.

Yeah, some are quite old and discontinued. I currently use Nightmare which is electron based and has a nice api. Tried a couple others but every time something went wrong.

The problem is that to take a screenshot of how the page actually looks, you need almost everything from the browser. Rendering of HTML/CSS/images/videos, Javascript execution, networking, cookies... you end up with 90% of the browser, and then removing the last 10% is not worth it.

If all you want to do is take a screenshot, I've had success with http://wkhtmltopdf.org/ (which also outputs to image formats.) You can't script it though.

The problem is that the "graphics part" is often the part that knows how to deal with fonts, which you sort of need for layout.

Here's a thing I made to script WebDriver from the terminal or shell scripts:

https://github.com/mbrock/wd

That's really cool! Like it a lot!

Thanks! I made it to automate some tasks involving saving receipts to files and so on. Using the shell as a REPL and being able to use the shell's functions for working with files and texts is why I like it.

Most of the tools listed use PhantomJS or QT core browser as the base. The problem is that PhantomJS doesn't support many new API-s or the rendering is different.

There is also headless chrome browser in development, not sure if there are any prebuilt binaries out there?

Going to ask same question, about headless Chrome, any recent information on that, anyone?

it is issue 546953[1] in the chromium issue tracker.

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=546953

There is a Google Group for it: https://groups.google.com/a/chromium.org/forum/#!forum/headl...

I check every now and again. I'm super interested in headless PDF generation (using Nigtmare now)

what new apis should phantomjs support?

download support[1] is the only new api in development. it's not tagged for the 2.5 release[2].

[1] https://github.com/ariya/phantomjs/pull/14225 [2] https://github.com/ariya/phantomjs/issues/14458

FileReader support is not fully implemented (addEventListener is not present). Just ran into this last week.

There has been an open issue for this for three years[1].

[1] https://github.com/ariya/phantomjs/issues/11733

I've been using chromium-headless for certain automated tasks, and I've been getting a lot better results than from PhantomJS in terms of correctly rendered resources, resource consumption and execution speed. If your business depends on rendering engines and you're using PhantomJS, chromium-headless is probably something you want instead.

I build from chromium src, and my code is more or less an extension with its own BUILD.gn with a deps on //headless. Once you've built the initial tree (took a night for me), builds will be really fast.

Most of the interaction with the browser is by the chrome debugging protocol, as with "normal" chromium. Generally speaking, you want to target that, instead of the internal APIs.

There are some tweaks you need to be aware of. Chromium needs a homedir, so you want to configure that. There's state stored between sessions, so if you want process invocations stateless, consider spending some time hacking or run the process in a fresh namespace/chroot/docker/jail/whatever each time. I had to write my own protocol handler and URLRequestJobs because I render pages from archives on disk and that was the easiest way to get that started at the time.

I don't know if vanilla chromium supports it, but I'm pretty sure I've seen a --headless flag somewhere... If not, headless_shell is pretty easy to build following the official build instructions[1]. I'm running on RHEL/Fedora/CentOS. There's also unofficial images on docker hub if you want to go that route.

[1]: https://chromium.googlesource.com/chromium/src/+/lkgr/headle...

Here's the bug to follow if you want to follow progress (the anchor comment in the url is for when screenshot capturing landed)

https://bugs.chromium.org/p/chromium/issues/detail?id=546953...

What about startup time? phantomjs is the only headless browser I have tested which starts up relatively fast.

Can't test now, but around a few tens of milliseconds for about:blank

Also selenium grid docker containers. Unfortunately their Python documentation isn't the best (though there's some excellent third-party created Python documentation available). To save you the head-scratching I went through, it's basically just:

- Start a 'selenium hub'

- Start up as many firefox/chrome 'node' containers and link them to the hub.

- Point webdriver at the hub port and url.

I will eventually get around to building some Alpine containers for this, one day. I find the official ones a bit 'heavy' given all it really invoves in installing openJDK and running a jar file / browser driver.

Anybody else seeing something similar to this? https://i.imgur.com/Dz1FLGQ.png. It's as if GitHub was embedded in an iframe on this repo's page.

Yup, I'm seeing the same thing. Weird.

Ditto. Probably someone messed up some piece of dynamically loaded server-side rendered stuff in the repository info bar. Since not everyone seems to have it it's probably not on full-roll-out, but just a small part of the user base.

Chrome will officially support headless mode: http://crbug.com/546953

The problem I found with this was mostly that there are bugs that only show in one browser (only FF, only Chrome, only Edge, etc) so these tools get their own share of bugs.

At the moment I use Ghostinspector (a SaaS) which works okay. Their UI is a bit clunky, but recording stuff via Chrome extension and modifying it later works okay. Also they let me run the tests with Firefox later. So I develop with Chrome and run tests with Firefox.

i've tried some of these for a project ( i had to cough automate download of files from a site made in old asp.net and with a lot of new js on it), only awesomium was able to do the job, but, man, it was overwelming difficult! i had to inject fiddler as proxy to catch exactly what was happening, awesomium by itself just can't do it, we are very far from a real scriptable headless browser.

Unfortunately, Awesomium seems to have been in "lights on, nobody home" mode for over a year now, and uses an outdated version of Chrome. [1] The nail in the coffin is a predictably unfulfilled promise (see Yannou's rule #1 and #2) followed by radio silence: [2]

"The product is very well alive. Version 1.7.6 will be released in a few days with many new features and an updated Unity plugin as well. Meanwhile, we are also working on our next major version (v2.0)." Perikles C. Stephanidis, Jun 15

The other big problem is that it's not free. So nobody is going to pick it up and run with it.

[1] http://answers.awesomium.com/questions/6880/does-the-project...

[2] http://answers.awesomium.com/questions/7255/is-awesomium-dea...

What about Electron? Multiple people, including myself, have used it to make a web browser

fwiw, nightmare is just a thin layer above electron's browser.

I'm currently using Nightmare for a little automation. It's fantastic. A few small gotchas, but so easy to set up.

Missing: lynx, links (both have headless mode), curl, wget, nscurl.

Pretty sure being able to render JavaScript is needed to be on the list. So not most of what you listed.

It's missing the browser engine and version that each of them use. Even latest PhantomJS only uses Chromium 40. Most important metric IMO.

isn't PJS based on webkit?

Same same

Would be nice to include if is it maintained or their last update. There are a lot of them that aren't updated in years.

I know it's somewhat a vanity metric, but this list should have GitHub stars listed for each project. Would save us some work :)

Unless someone creates a bot that will automate the process of updating the number of stars, it's a cat and mouse game.

Maybe something like this would be an easy start: http://githubbadges.com/

number of contributors, last commit date too.

Reply on news.ycombinator.com