Hacker News Reader

Comment by bluepnume | Hacker News Reader

This is fantastic. I'm using a combination of Chrome and PhantomJS for karma testing right now, for https://github.com/paypal/paypal-checkout and https://github.com/krakenjs/xcomponent. There are hundreds of tests opening up hundreds of iframes and popup windows, and sending a lot of cross-window messages, and that ends up being really memory hungry.

Chrome deals pretty well with garbage collection, so long as I'm careful to de-reference closed windows properly¹, and only uses a maximum of 150mbs. PhantomJS eats up almost 6GB of memory before it's done, which makes it almost unusable on machines with less memory or CI boxes. Travis is a no-go.

I'm hoping running Chrome in headless mode should give a nice speedup for our tests.

-----

¹ Turns out even a closed popup window or iframe keeps a huge amount of memory hanging around. Who knew.

View on HN

We (Chrome) have reached out to PhantomJS to inquire if they're interested in collaborating: https://groups.google.com/d/msg/phantomjs-dev/S-mEBwuSgKQ/tU...

The DevTools Protocol is the primary API for headless Chrome, but we are excited for higher-level abstractions like PhantomJS & NightmareJS's API to manipulate the browser as well. Plenty of details to work out, but hopefully sometime this year you'll get a drop-in solution for some of your testing to upgrade from Phantom's older QTWebKit to the latest Chromium.

Thank you for being willing to work with them and not just rolling your own version of Phantomjs. I hope they respond positively to your inquiry

It was my understanding that Google's crawler literally is Chrome. Does Google have any plans to open source those parts of the browser to make integration easier? Maybe I was mistaken

That seems unlikely... It would seem they have two sets of crawlers... one that does typical/advanced scraping, and another that runs the JS (Chromium) and takes DOM snapshots. This is reflected by changing certain properties (window title etc) and seeing them reflected in Google's search results. A couple years ago, the lag was several days to a week behind on new content via as-rendered from the server vs. via JS.

Interesting! I find it puzzling that insider knowledge on how google search works never seems to leak into the public domain. Is google killing leaks in their search algorithms or do they pay the search team such ungodly amounts of cash that nobody has ever left?

Well, the point of their engine is to make it harder to game the system... everytime someone figures out a trick, someone takes advantage of it... that alone would discourage leaks.

We laypeople can only find out things via word of mouth or observational tests on assumptions.

> The DevTools Protocol is the primary API for headless Chrome

Why not the WebDriver protocol? That seems to be exactly what it's intended for…

WebDriver protocol has limited APIs. For example, no networking status. (WebDriver rejected to implement it, they told you to use Proxy for debugging)

A few hours later: https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuN...

After his announcement, Vitaly answered to the team: https://groups.google.com/d/msg/phantomjs-dev/S-mEBwuSgKQ/PQ...

> >Is there interest on your side in adopting Chromium as a runtime? There's some existing documentation [2] around the API and embedding, but admittedly, this would be some work.

> We are interested. But I am afraid not in the current state. Currently, PhantomJS heavily relies on Qt and QtWebKit. It's not that easy to adopt Chrome as a new runtime.

> But I think we could implement PhantomJS as a completely new (with the same API) project that will use Chrome - Phantomium!

Also see https://github.com/Vitallium/phantomium

We're using xvfb and selenium* for testing and a proper headless support would be a hundred time more stable than the self restarting framebuffer can't wait to move to headless chrome

*Yeah I know phantomjs is more cool these days but phantom doesn't support windows height so there's that.

+1 for xvfb

We also did a POC with PhantomJS and found similar issues, as well as generally flakiness causing too many false negatives. Ended up not using it; I'm hoping this can simplify things are give something more solid to build on.

Same, ended up using ChromeDriver + XVFB on Jenkins. Works but it's still pretty slow.

I replied to you on Twitter, but for the sake of discussion, there is a third option that may be worth exploring - using Chrome running in virtualized Windows Manager - https://www.alexkras.com/running-chrome-and-other-browsers-i...

Very cool. Gonna have to check this out.

Hello, former SDC coworker!

The official Selenium Docker image uses the same technique to run headless Chrome / Firefox.

From there you can just run

  docker run -d -P selenium/standalone-chrome

and you'll get something a lot more lightweight than spinning up a new Vagrant VM.

What do you do to ensure the resources for closed windows are released?

A combination of:

- Making sure all promises are fulfilled or rejected, so window objects don't get caught indefinitely in closure scope for any .then() or .catch() handler functions.

- Using WeakMaps as much as possible, when we have things that are tied to a particular window, like message listeners or response handlers in post-robot

- Manually clearing up any global references to windows when we destroy an xcomponent instance

Finding the references was the tricky bit. A lot of the effort was finding a leaky test-case, running in 100 times in succession, and deleting code until the memory graph was flat -- then figuring out what I'd just deleted that caused the leak.

The problem started manifesting as I added more and more tests -- so now I'm actually checking my tests' memory usage on the fly and failing if they cross a threshold. Hopefully that should avoid getting into this kind of sticky situation ever again.

https://github.com/paypal/paypal-checkout/blob/master/test/t...

I've been using post-robot and xcomponent a lot recently as I'm solving a bunch of similar challenges. Just wanted to say thank you for sharing your solutions and expertise.

Glad they're working out for you! Let me know if you encounter any interesting situations they don't support.

But can you run headless chrome on Chromeos? if not why not?

Reply on news.ycombinator.com