Show HN: Automatically create QA environments for your GitHub pull requests | Hacker News Reader

Show HN: Automatically create QA environments for your GitHub pull requests

prune.io

72 pointsmpotter20 comments

Dev here. Our team has grown to love the GitHub pull request workflow. We built Prune to solve one issue we consistently run into: how to QA on the various branches before merging. Happy to answer questions and thanks for any feedback!

looks awesome

Can you give more details about Visual Regression Testing? If you're comparing screenshots, how do you account for dynamic content?

Sure! We use a web driver to simulate browsing in an actual client, so Javascript, etc. is executed as it normally would be.

But what about content that changes on each page load, like a timestamp or a rotating banner ad? Would that be flagged as a regression?

Yeah, it's not perfect but right now we're solving this with tolerance thresholds on the image diffs.

Can you share any more details on how the image diffing is done? BBC has a github repo called Wraith that uses ImageMagick and PhantomJS to accomplish a similar task (without the awesomeness of on-demand testing environments). Always curious to learn more about how people are solving the GUI testing problem on the web.

There's also Depicted (dpxdt)[1] - more workflow, Python instead of Ruby.

[1] https://github.com/bslatkin/dpxdt

WebDriver/Selenium? That's not really visual, like some kind of CV/OCR tool like Sikuli would be. It's a nice tool, I would be more inclined to use it if it supported more robust test environment provisioning (eg, if the pull request is a schema migration) and it gave more control over how to define my own tests, like Sauce Labs.

edit: never mind, I realized you're probably using the built in screenshot features with webdriver and just diffing the images.

I just wanted to know how this works. Our company uses a pretty neat setup where we run different different branches under a wildcard domain on a single machine.

Basically, we run a *.staging.domain.com; however, databases are always tied together. (This is for Wordpress purposes). We use [Genesis](https://github.com/genesis/) to achieve this effect.

So, do you guys work similarly to Travis-CI in that you run an instance, follow instructions in a file and how to provision and setup the site, and then instead of just running a unit test (like travis), you keep the instance running and expose it to public?

Do you use virtualization or docker instances? (Just curious!) How long do you keep them up and do you suspend the instances when it becomes disused? Will you be able to run a single instance at a time or many? (meaning, one branch at a time or will you have 20 environments setup for my 20 pull request).

In the future, will you support private repos?

Thanks! This looks REALLY badass! :)

"Who are you people?"

Why are you committing before making a feature branch, is this part of some flow? (just surprised me a bit) This is as sick as automated CI gets though.

That's pretty nice, although... since there are already environments to run tests (e.g. travis) why run them again in Prune, and why not wait until the QA environment is setup to display the notification?

I dig the visual regression thing, though I fear combinatorial explosion for big-ish changes if it goes through the whole site or application.

I just set something like this up for a project I'm working on, using jenkins and (of course) docker. The level of project integration required makes me wonder how easy it is to offer it as a service, but I guess if you can solve that problem you'll get customers.

Neat idea.

What's the underlying technology creating the QA environment? A virtual machine? A container? How is it being configured? A CM tool?

Go-go-gadget Docker. (according to mpotter below)

This looks awesome. I love the auto screenshot feature.

How do you manage to support whatever crazy environments a dev may cook up and all the services needed to get a working QA environment set up?

We'll announce our supported environment list shortly but the preview answer is: Docker & Heroku Buildpacks :)

This looks great - Interested to see the pricing - I'd want to use it across all of our clients projects.

I want this.

Reply on news.ycombinator.com