Hacker News Reader

Comment by joefarish | Hacker News Reader

It's obviously not a headless browser but xvfb can be very useful depending on what exactly one is trying to test: https://en.wikipedia.org/wiki/Xvfb

View on HN

Xdummy is another headless X display that has a more modern, more complete stack.

https://xpra.org/trac/wiki/Xdummy

TIL. I did not know Xdummy. I have just followed your link. It is trivial to launch. It avoids installation of vnc, has a better compatibility and is already present everywhere.

This is the way I am going.

I spent a long time trying to find an up to date javascript engine in a headless browser a while back before I gave up and just wrote a chrome extension, installed a copy of chromium, and ran the whole thing in xvfb with the extension added by command line.

In the end unless you are severely resource constrained,it is a very good option.

I've done a similar thing. There are annoyances with both XVFB and Firefox instances clobbering each other though. For XVFB, each instance needs a globally unique display number. For Firefox, you need the "-new-instance" or "-no-remote" option, along with a unique profile (e.g. in a temp dir).

Could you say more about what you mean by "clobbering each other" here?

I run firefox/chrome under Xvfb and very frequently get unexplaine `Unable to connect to host 127.0.0.1 on port 7055` from selenium, zombie firefox and chrome processes, and hundreds of TCP ports in status TIME_WAIT.

I've always assumed it had something to do with Xvfb/browser interaction.

Firefox shares sessions (cookies, etc.) between all running instances by default.

When you run "firefox", it will check if the current user already has an instance running; even if it's on a different display.

To prevent that you need to run the firefox command with "-new-instance" or "-no-remote" (which implies "-new-instance").

Firefox also 'locks' profiles which are in use, so you need to use separate profiles for each Firefox instance which is running. You can do that by making a temporary directory "T" and passing it to the firefox command as "-profile T".

As an example, the Firefox command I'm using in one of my scripts is currently:

    timeout 60 firefox -safe-mode         \
                       -profile "$FF_DIR" \
                       -no-remote         \
                       -new-instance      \
                       "$URL"

XFVB is an X server, which programs connect to based on the "DISPLAY" environment variable. You can see what that's set to by running:

    echo "$DISPLAY"

You can set it, e.g. to ":99", by doing:

    export DISPLAY=:99

Each running X server needs a different display number; this includes regular desktop sessions (usually running on Xorg with display :0), XVFB, Xnest, Xephyr, etc.

I use a variant of the script at http://stackoverflow.com/a/30336424/884682 to choose an unused display number for each XVFB invocation.

NOTE: This is based on simple automation tasks with bash and xdotool. I don't use Selenium, so I don't know if already does any of this. Hope it helps anyway :)

Just to throw in another option, there's selenium grid docker containers: https://github.com/SeleniumHQ/docker-selenium. The containers are a bit heavy IMHO (I suppose because they're aimed more at 'testing' instead of scraping), but I think there are a few Alpine Linux versions floating around (and easy enough to set up yourself).

Handy for more tricky sorts of web scraping, or if you want to use chromium in mobile emulation mode (causing websites with an 'app version' to switch over to a HTTP-JSON SPA in many cases). I was particularly excited to discover you can set the 'http_proxy' env variable on a "per-browser node" basis (and do fun stuff like installing DNS caching daemons, round-robinning through OpenVPN/squid containers etc.).

+1 you can run things in real browsers. fwiw Travis CI supports it.

You can also go with a regular browser tied on a vncserver display (pros: you have real captures/video of the tests, cons: the solution gets a bit heavier than when using xvfb)

It's also fairly easy to take a capture of an xvfb or an xdummy display. Dunno about video though.

Reply on news.ycombinator.com