Ephemeral Environments for DevOps: Why, What and How? (2017) | Hacker News Reader

Ephemeral Environments for DevOps: Why, What and How? (2017)

enterprisedevops.org

40 pointsroyosherove14 comments

Shameless plug: I'm currently working on a service that'll spin up / tear down environments automatically for each branch in your repo. We're focusing on the manual testing use case between designers and front-end engineers, so prod deployment is omitted from our featureset. But if you're looking to spin up ephemeral environments without building something yourself, check us out: https://featurepeek.com

Great stuff, I like the idea of bringing the same level of reasoning that's been hard learned over the past 50 years of software development to devops too.

Currently the state of devops tools includes omitting debuggers when a company makes a new DSL, poor if any linting, low to non-existent test coverage so obvious bugs and regressions are normal, very poor IDE integration if any, often no real way to test locally, and not surprisingly the tools are bug riddled and at best scratch a single itch in an eco system w/o directly addressing the overall business problem.

Our product EnvKey[1] can also help to secure and simplify configuration of ephemeral environments—all you do is set a single ENVKEY environment variable, include an envkey library, and then all the secrets/credentials/config you need will be made available and kept in sync on each load. You also get a nice ui for editing variables and managing access. Ah, and all this is done without trusting a central server through the magic of end-to-end encryption. I think it could potentially jive well with a tool like yours given that proper secrets management can be a tricky piece of this puzzle.

1 - https://www.envkey.com

This is really interesting. My team is currently solving the question of ephemeral env vars through a lot of wololo (for you Age of Empires players out there) in terraform and PUT-ing the results into a local consul cluster.

Being able to instead write an env and have my application not particular care where it retrieves it from as long as a library is available is something I'm going to put in front of the rest of my team on Monday.

The primary problem with ephemeral environments isn’t distributing secrets, it’s provisioning the environment itself. It’s perfectly fine to generate an empty, self contained, with dummy “abcd1234” passwords or randomly generated credentials that are used only locally.

That works for services you control, but if you depend on external services, and these are relevant to what you’re attempting to test with a tool like FeaturePeek, then you still need to deal with credentials. Or am I missing something?

Stumbled upon this the other day! Will take a look

Currently my team uses Heroku review apps for this, and the experience has been excellent. Opening a PR for our frontend automatically opens a review app, which gets designer sign-off before code review.

So, as a dev-ops engineer, I'm trying to decide if I'm onboard with this idea. I still don't see the justification though.

The idea of provisioning an environment to test a branch seems like a lot of overhead. Concretely, how long does it take to spin up an environment, install the containers/app, and run tests?

I imagine the answer is going to be over an hour.

Definitely feels like a lot of overhead, particularly when most branches change ~.1% of an application's code and so theoretically the full-diff is kilobytes worth of data.

The alternative of course is just to have a pre-set number of build-servers (e.g. 5) and then have jenkins dynamically run jobs against them. Seems simpler, I assume faster, and just less risky.

> The idea of provisioning an environment to test a branch seems like a lot of overhead.

The idea of running integration tests on a non-complete environment seems like a lot of fun at the release.

> Concretely, how long does it take to spin up an environment, install the containers/app, and run tests?

I work currently at the community-sized online bank, total time of our pipeline to spin up a new "bank" is 22 minutes, running multiple of these envs on 4 HDDs in RAID 10. We'd move the whole thing on flash array in 2 weeks and my estimation is that this metric will go down to 7-8 minutes.

> The alternative of course is just to have a pre-set number of build-servers (e.g. 5) and then have jenkins dynamically run jobs against them.

You have to run these builds somewhere, if changes would run on a single staging env, it would be down most of the time. If it would be a handful of stagings, it might be still not enough because either you wouldn't have enough of them and it'll ended up to the single env situation multiplied, or you would burn cash for idle servers.

We need this so much.

The environment is part of the product. With software, if you can't easily and automatically recreate your builds from source, you have a problem.

This ought to apply to cloud environments too.

Wow is this ever overkill.

This needs to be coupled with a lot more “how and we we picked this methodology” advise or information.

For example, if the goal is to reduce risk then is this the best way to approach that? Are all these components black boxes which cannot be stubbed out? Are there any other end to end or integration tests that can be ran in a canary environment that are way faster which will cover the most obvious use cases?

Id really think a few times before adopting this practice.

Currently DevOps tools do almost no testing or validation at all, it’s like time traveling back to the 80s.

Internal validation of a backend system could be automated but that would be heck of a job, anything beyond that requires human to interact and report.

Reply on news.ycombinator.com