Hacker News Reader

Comment by olalonde | Hacker News Reader

On a related note, is anyone aware of any tool(s) to declaratively describe an application architecture and automate its deployment on something like AWS or in a single virtual machine + docker? For example "I want 10 replicated microservice1 processes and 5 replicated microservice2 processes in front of a nginx load balancer. Also, throw in a Redis and a PostgreSQL database." This would be immensely useful for (one man show) developers who don't necessarily have the time or experience to "properly" setup and fine tune infrastructure. I think what I'm describing is known as "Infrastructure as code" in DevOps parlance.

Edit: Since this question is getting many replies, I'll be a bit more specific in what I am looking for. Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes.

View on HN

Cloud Foundry + BOSH, IME, does exactly that.

http://bosh.cloudfoundry.org

https://github.com/cloudfoundry/bosh-lite

https://github.com/CloudCredo/dockpush

You download a BOSH release for your infrastructure, tweak the manifest for your chosen cloud, and push the deployment out. If you make a change, you just push it out.

Cloud Foundry itself (the elastic runtime) is a BOSH application that includes an HAPRoxy load balancer, dynamic router, and allows you to easily push/scale the stateless micro services and declaratively wire them up with other services you've deployed (with CF or Bosh).

It can run on AWS, vSphere, OpenStack, or on your own Vagrant VM in containers (bosh-lite). Or you can use Pivotal Web Services public cloud: http://run.pivotal.io

BOSH requires some wrestling early on but I find it to be a fascinating project.

We built that at a previous employer:

https://github.com/tim-group/stackbuilder https://devblog.timgroup.com/2012/12/20/towards-a-rich-domai...

The documentation is pretty rubbish, but here's an example config file:

https://github.com/tim-group/stackbuilder-config/blob/master...

It uses libvirt for provisioning, and Puppet for configuration. The coupling is extensive but not tight. It's also coupled to a specific way of deploying Java apps. There's a sort of backdoor that would let you deploy other kinds of apps by writing some Puppet code.

I do think the basic ideas are good, though. Possibly more of an object of study than a tool to actually use!

Are you looking for something like https://www.tutum.co maybe?

Locally, you'd use http://www.fig.sh to describe and run your app. You can then import this description into Tutum and run it on e.g. DO or AWS.

> Edit: Since this question is getting many replies, I'll be a bit more specific in what I am looking for. Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes.

Seems like the https://www.hashicorp.com/ guys would have something, perhaps Terraform/Atlas?

I have not done this yet obviously but am in the same boat as you for our company goals this year

What I liked most about Terraform was the explicit separation of planning and execution.

Cloud66 can handle most of what you describe using manifest files. You can have a relatively simple yml file describing what types of servers exist in your stack - something like "I need Postgres servers, Redis, a RabbitMQ server, oh and here's the 3 different types of process servers I have and also how my main web server functions."

There's still a point where it's beneficial to split an application into seperate stacks, in which case the manifest concept breaks down and you're stuck doing a lot of work yourself, but they can take you a fairly long way.

> There's still a point where it's beneficial to split an application into seperate stacks

Do you mean like having a different "infrastructure manifest" for each micro-service? In that case, couldn't each micro-service specify which other micro-service it depends upon? For example, I could have a central "API gateway" service which would specify all the other micro-services it depends upon. This reminds me a bit of a Gemfile/Package.json but for processes instead of libraries. On the other hand, I can easily see how this could turn into a dependency hell.

We're working on a very low-tech version of this at Zalando, here: https://github.com/zalando/aws-minion

Sounds like you want AWS OpsWorks? https://aws.amazon.com/opsworks/

NixOps: https://nixos.org/nixops/

Yeah, NixOS + NixOps declaratively define logical and physical part of an machine network.

Docker with CoreOS and fleet will get you pretty far on that front. You can just specify that you want 6 of container x running across the cluster, and 3 of container y.

Supplementing this with Hashicorp tools like Terraform and Consul can get you to the last step.

Salt/salt-cloud operates as you describe. It allows you to define any number of virtual machines associated to a cloud provider 'profile', i.e., instance type, size, platform, etc.

There are some limitations still, but ubuntu juju [1] was the first thing that came to mind that somewhat fulfills the requirements.

[1] https://juju.ubuntu.com/

For OpenStack clouds, the HEAT project is working on that.

> Is there a tool that would let me describe the infrastructure and deploy it on a given cloud provider but also have the ability to deploy the same infrastructure on my local machine (using VMs/docker) for development purposes.

Depending on your concept of "deploy", I can point you to two options, both produced by the company I work for.

If you think in terms of controlling from the VM up, look at BOSH[1]. It's an IaaS deployment / management / update tool which has been used in production for several years.

You create two yaml files: a release manifest (describing the software components of your system) and a deployment manifest (describing the computers and networks of your system and how to map components to them).

This separation means that a single release manifest can be applied to many different actual deployments. At my day job I deploy a large PaaS release into a single virtual machine. My colleagues deploy exactly the same release into AWS with thousands of machines.

Systems defined using BOSH can be deployed to AWS, vSphere, OpenStack or Warden (a containerisation system).

For local development, the Warden backend enables "BOSH lite"[2] -- releasing and deploying into a virtual machine that hosts a local cloud of containers.

If you want to go the next level up to a proper PaaS, you can try Cloud Foundry[3]. It's defined as a BOSH release[4], which is how at my day job my co-workers and I are able to work both locally and at scale on identical software.

In Cloud Foundry we distinguish between stateless (apps) and stateful (services) software, and we provide a simple way to connect them. The easiest way to play with it is to log into Pivotal Web Services. IBM Bluemix is the same software and is also open to the public.

So for example, with Cloud Foundry, if you want ten copies of your app, you do this:

    $ cf scale my_app -i 10

And it'll be scaled up to 10 instances.

Need more RAM?

    $ cf scale my_app -m 2G

Now each node gets 2Gb.

And so on.

[1] https://github.com/cloudfoundry/bosh

[2] https://github.com/cloudfoundry/bosh-lite

[3] http://docs.cloudfoundry.org/

[4] https://github.com/cloudfoundry/cf-release

Reply on news.ycombinator.com