Hacker News Reader

Comment by mpotter | Hacker News Reader

Dev here. Our team has grown to love the GitHub pull request workflow. We built Prune to solve one issue we consistently run into: how to QA on the various branches before merging. Happy to answer questions and thanks for any feedback!

View on HN

Can you give more details about Visual Regression Testing? If you're comparing screenshots, how do you account for dynamic content?

Sure! We use a web driver to simulate browsing in an actual client, so Javascript, etc. is executed as it normally would be.

But what about content that changes on each page load, like a timestamp or a rotating banner ad? Would that be flagged as a regression?

Yeah, it's not perfect but right now we're solving this with tolerance thresholds on the image diffs.

Can you share any more details on how the image diffing is done? BBC has a github repo called Wraith that uses ImageMagick and PhantomJS to accomplish a similar task (without the awesomeness of on-demand testing environments). Always curious to learn more about how people are solving the GUI testing problem on the web.

There's also Depicted (dpxdt)[1] - more workflow, Python instead of Ruby.

[1] https://github.com/bslatkin/dpxdt

WebDriver/Selenium? That's not really visual, like some kind of CV/OCR tool like Sikuli would be. It's a nice tool, I would be more inclined to use it if it supported more robust test environment provisioning (eg, if the pull request is a schema migration) and it gave more control over how to define my own tests, like Sauce Labs.

edit: never mind, I realized you're probably using the built in screenshot features with webdriver and just diffing the images.

I just wanted to know how this works. Our company uses a pretty neat setup where we run different different branches under a wildcard domain on a single machine.

Basically, we run a *.staging.domain.com; however, databases are always tied together. (This is for Wordpress purposes). We use [Genesis](https://github.com/genesis/) to achieve this effect.

So, do you guys work similarly to Travis-CI in that you run an instance, follow instructions in a file and how to provision and setup the site, and then instead of just running a unit test (like travis), you keep the instance running and expose it to public?

Do you use virtualization or docker instances? (Just curious!) How long do you keep them up and do you suspend the instances when it becomes disused? Will you be able to run a single instance at a time or many? (meaning, one branch at a time or will you have 20 environments setup for my 20 pull request).

In the future, will you support private repos?

Thanks! This looks REALLY badass! :)

looks awesome

Reply on news.ycombinator.com