Hacker News Reader

Comment by stevebmark | Hacker News Reader

I haven't heard of using docker compose for production development, only local development. In production I normally see people building production images and spinning up them up in their cloud with Terraform and whatever other infrastructure they need, especially for databases where your local MySQL container isn't going to be configured or provisioned like an AWS RDS instance. What do you hand a docker-compose file off to in production? Swarm? When would you choose this path over a separate system to provision your production resources (like Terraform)?

View on HN

OP here.

A lot of my clients are happily using Docker Compose in production. Some even run a million dollar / year businesses on a single $40 / month server. It doesn't get handed off to Swarm or another service, it's literally docker-compose up --remove-orphans -d with a few lines of shell scripting to do things like pull the new image beforehand.

You can still use Terraform to set up your infrastructure, Docker Compose ends up being something you run on your server once it's been provisioned. Whether or not you decide to use a managed database service is up to you, both scenarios work. The blog post and video even cover how you can use a local DB in development but a managed DB in production using the same docker-compose.yml file, in the end all you have to do is configure 1 environment variable to configure your database at the app level with any web framework.

how do you handle no downtime deployments? how do you backup data?

I rather would use kompose+kustomize to convert it to k8s and than use k3s with ingress-nginx with hostPath and backup local volumes with velero.

You don't always need "no downtime deployments", few minutes are tolerated in many environments (I work in enterprise software, where it could be days!).

Backups? You only need backups or the Db, and that have zero impact in the use of docker or not*

*ie: I know what very complex setups the micro-service crow use for serve their smalls-size disruptive app. I still think cron + pg_dump + rsync is more than enough...

well I'm asking because I have a docker-compose file and I use kustomize+kompose on it, because volume backup is a thing and the file comes from an upstream vendor and we can actually miss a few minutes/hours/days of uptime, but since the databases also use compose and sadly in a way that I can't remove it (migrate.sh script that makes use of compose commands), else I can use my kustomize+kompose setup which basically replaces the kustomize+kompose with a custom init container that I built, however everytime they change their shell scripts I need to fix my stuff... but they say that I should just backup my whole volume/vm via lvm snapshots or via vmware whatever. which I find a little bit akward and I do not want to backup my data with disk snapshots. another way would be to natively backup stuff via compose, but they use many databases and it's way harder than just use velero.

I’m not the parent but I’ll pitch in on this as I have a few client projects using docker compose for production.

No downtime deployments: I don’t bother for these projects. A minute of downtime in off-peak hours for a restart is fine.

Data backup: just like you would backup any other local data: cron, rsync, pg_dump. And there isn’t much local data anyway as most services are stateless, using S3 or a database for long term storage.

I choose docker compose for production when I want to keep complexity to a minimum: All I need is a cloud vm, an ansible playbook to install docker and configure backups. Typically in these scenarios I run the database as an OS service and only use docker for nginx/redis and application services.

Why do you use the database as an OS service and not in docker?

Using it in docker would allow easier local development, and shorter setup time.

For local development I run the full environment including the database in docker compose.

For production some of these projects run alongside each other on the same host and share the database service. Sometimes the database is on another host.

A typical host like this would be a 50/month Hetzner dedicated machine with only docker daemon and postgres installed natively. And then multiple projects deployed with docker compose. Sometimes nginx is native as well with vhosts pointing to the docker projects.

Edit: s/ghosts/vhosts/

It is really not recommended to run (production) databases in Docker, unless you know what you are doing.

https://vsupalov.com/database-in-docker/

We've been doing this because alternatives like k8s burned out time, $, and other team members, and our on-prem customers like it operationally vs k8s. SW teams ran high uptime stuff before k8s, and you can still do most of those tricks. We're revisiting, but selectively and only b/c we're now getting into trickier scenarios like multi-az and potentially elastic GPUs for our SaaS offering

Ex:

- blue/green stateless nodes

- DNS/load balancer pointer flips

- Do the DB as a managed saas in the cloud and only ^^^ for app servers

- Backups: We've been simply syncing (rsync, pg_dump, ..) w/ basic backup testing as part of our launch flow (stressed as part of blue/green flow), and cloud services automate blob backups. We're moving to DBaaS for doing that better, which involves taking the DB out of our app for our SaaS users

In practice, our service availability has been largely degraded by external infra factors, like Azure's DNS going down, than by not using k8s/swarms for internal infra. Likewise, our own logic bugs causing QoS issues that don't show up in our 9's reports.

> We've been doing this because alternatives like k8s burned out time, $, and other team members, and our on-prem customers like it operationally vs k8s.

It doesn't seem you bothered looking into alternatives.

Take for example docker swarm. It already uses docker compose files (well, there are some differences) , they support blue-green deployments of stacks, and they even support multi-node deployments and horizontal scaling.

Swarm was/is a bad bet due to looking like Docker gave up on it, esp. for deploying to our on-prem users where we can't just change our mind the next day, and wanting something we'd have some faith in for 2-3 years. Nomad was more appealing from the simplicity view, esp. as we'd need to do some rewriting either way.

> I rather would use ...

you'd rather use six complex things instead of one simple one?

well I asked, because everybody answered either with the following:

> we do not care about no downtime or stateless nodes and a lb in front

and b:

> backup with the standard tooling or stateless nodes

so basically they just use it mostly to "package" the application and everything else is probably done with different tooling.

> you'd rather use six complex things instead of one simple one?

By that measuring stick, using Docker Swarm instead of docker compose is even simpler as you only need to use docker alone instead of having to install a separate script.

docker-compose is now built into the docker command line. e.g., `docker compose` instead of `docker-compose`

How's it different than running things with pure docker or before docker? There were backups and no downtime deployments before docker became a thing... docker compose is already a layer above docker (but really just docker).

yeah but stuff like that is hard and not solved with compose, you can update your compose but that would result in downtime, so you can either do a/b (blue/green) deploys or you somehow reload the content inside the container?! everything that needs special tooling.

You can have a compose with multiple backend services and an nginx load balancer in it proxing back to it each one, then down/up each each backend service individually when upgrades for example..

Which may force you to handle two versions of databases schemes instead of a single migration. This adds complexity and sometimes 1min of down time is more acceptable than that additional work.

Yeah for sure, but I feel that's a software design problem and not directly related to using docker compose or not, you could have this same issue regardless of how you go about deploying it.

> You can have a compose with multiple backend services and an nginx load balancer in it proxing back to it each one, then down/up each each backend service individually when upgrades for example..

Things are way simpler with Docker Swarm. You just run a single ingress controller as a separate stack and route all traffic internally.

If you use Traefik as your ingress controller, all you need to do is set labels in your docker compose files and everything just works.

There is absolutely no reason or excuse to use docker compose in anything resembling production.

> There is absolutely no reason or excuse to use docker compose in anything resembling production.

I work at a university where I make an app meant for maybe a couple of concurrent researchers. Am I seriously supposed to set up Docker Swarm for this or is it OK with you if I run my little setup using Docker-compose on the tiny server that the university has provisioned for me?

Seriously, why do so many people on HN have to act like the entire world of software developers is busy making the next Facebook? I'm sure many (most?) people are making tiny things meant for tiny audiences.

> Seriously, why do so many people on HN have to act like the entire world of software developers is busy making the next Facebook? I'm sure many (most?) people are making tiny things meant for tiny audiences.

I agree generally, but playing devil's advocate for GP, what I think they meant is that swarm can be run on a single node with little downside/complexity and a lot of upside from docker compose.

See for example:

https://jarredkenny.com/single-node-docker-swarm/

https://devopstuto-docker.readthedocs.io/en/latest/docker_sw...

Thanks for the links. I was not aware that it could be run on a single node. Still, the benefits are unclear to me. I guess I will research it some more.

> Still, the benefits are unclear to me.

Manage deployments as stacks, which support history and backtrack deployments.

Blue-green deployments.

Frankly, it boggles the mind how people put to production devtools that are barely managed to be secure when deployed locally, when alternatives are both simpler to deploy and more capable.

The charitable assumption is that people use the (best) tools they're aware of. Discovery can be really hard today.

I am in charge of every single thing from devops to data modelling to backend web services to frontend apps to design and testing to responding to inquiries from users of a dozen web services. Does that boggle your mind too? That is the reality of my job. I don't have the time to be an expert in every single area when I am the one who has to do everything.

I'm sure you're great at setting up servers. That is probably the part of my job I dislike the most, but it's something I have to do, so I do it.

Docker has documentation on how to deploy compose apps to services like Azure Container Instances or Elastic Container Services. It's specifically for using compose in production.

https://docs.docker.com/cloud/aci-compose-features/ https://docs.docker.com/cloud/ecs-compose-features/

That looks much more complicated than scp a yaml file and run a single command. Which is all you need for a lot of real production apps.

Another one here! I use docker-compose, normal periodic DB backups to S3 and a twilio script that can use a lambda function to reload the entire system or parts of it by simple phone commands when necessary. Site does millions of views per day.

Nice. Reminds me fondly of many years ago when I sent commands via pager to long-running jobs running at work from the comfort of a nearby stool at the pub. Was eating dinner of course. ;-)

> how do you handle no downtime deployments?

I take the few second downtime hit on each deploy. I know, it's iNsAniTy but it's really not. A 5 second blip a few times a week is no biggie for most SAAS apps or services.

To me that's well worth the trade offs of not having to deal with a load balancer, a container orchestrator tool, dealing with database migrations that need to be compatible for 2 versions of your app as it's being rolling restart and all of the other fun complexities of running a zero down time service.

You can still architect your app in a flexible way (env vars, keeping your web servers stateless, uploading user content to S3 or another object store, etc.) so that moving to something like Kubernetes is less painful if / when the time comes. It's just for me, that time has never come.

> how do you backup data?

A background job / cron job runs every few hours and does a pg_dump to a directory in a block storage device. Could easily change that to be S3 too. Just comes down to preference. Personally I found it easier to drop it into block storage since it's just copying the file to a directory.

Your pragmatism is refreshing. This simple solution meets all your requirements so there isn’t a need to add complexity. Kudos!

Thank you for a sane comment! In many cases, downtime can even be contractually ok.

Nice and simple, love it

This is a great response and refreshing to see things can just be simple, if one tries to cover 99% of usecases and not go crazy overboard trying to solve just 1% of edge cases with "container orchestration" :)

I like this pragmatism and I teach it as well (https://deploymentfromscratch.com/). Just want to say that you can employ systemd socket activation for systemd services, and you can have zero-downtime deployment without load balancers.

I know this wasn't intended as satire, but that's how I read it.

Very modern stack but totally overengineered.

How do you do any deployments?

It's common to have a load balancer, that routes traffic to nodes that are up.

That part is mostly straightforward. It could be the application's responsibility or the load balancer's. The tricky bit is maintenance on the load balancing parts.

I've done zero-downtime deployments with the same Docker compose file running on multiple nodes. I just had a Python script (~100 lines) that would take instances off the load balancer in turn, update the docker-compose file, add them back, wait for a warm-up period and move on to the next ones.

Do you bother with linking your docket-composed services with systemd, for coordination during reboot,etc?

As long as the docker daemon starts on startup, there's a `restart: always` option in the docker-compose.yml config file that takes care of it. Using `depends_on: [list]` takes care of startup order cleanly.

All of the example apps in the blog post use restart: "unless-stopped" which ensures they come up automatically after a reboot or if they crash but can be recovered.

Then in dev the restart policy is set to no. This is handled with an environment variable.

That's my preferred workflow for most internal services and things at home, and I got annoyed that there was nothing to automate them with, so I wrote Harbormaster:

https://gitlab.com/stavros/harbormaster

You give it a config file with a few repos and it pulls/restarts whenever one changes.

This looks super interesting, I'm looking forward to digging into it more.

Thanks! Do let me know if it works for you, I love it so far but I've only deployed it in a few places.

It's not clear to me from the README whether harbormaster will keep up to date with docker images in docker repositories as well. How does this work? Meaning to say, when there is a change to the repository, where do the new docker images come from?

Use case is when a repository requires compilation to build the docker image, and I don't want that happening on the production server.

Ah yes, sorry, I should clarify this in the README: Docker images should be tagged in the repo (so you know what you're deploying), and Harbormaster will deploy the new image when you update the tag in the git repo.

It doesn't currently check upstream Docker registries for changes like Watchtower does, mainly because I think that's not a deterministic enough way to do deployments (but that can certainly change).

> What do you hand a docker-compose file off to in production

Why hand it off to anything?

> When would you choose this path over a separate system to provision your production resources

When you have a simple system with modest availability requirements and you don't like unnecessary complexity.

I don't think you would in prod, unless you were happy with it all running on one host.

Shame really, I find compose simple and neat and k8s+terraform an unsettling mess.

Docker swarm deploys to multiple hosts and there is also kompose for k8s.

I haven't used either of them in production though

The single host model is really pretty common

Personal or small/medium business sites, research software, free web utilities

What's considered lean in startupland can get pretty beefy compared to the shoestring a lot of projects are compelled by necessity to get by on

Compose can be great if you just need a web server and maybe a database and background worker and you don't have anybody suing you or losing millions for the occasional maintenance hour

Totally agree, wasn't slating the idea at all, simplicity wins out nearly every time for me.

It is an argument I've lost with managers more times than I've won though.

A single powerful host can run a quite large business; you might put two identical instances of that host behind a loadbalancer to reduce downtime due failures or upgrades/deployments, and for a single pair of hosts complex orchestration is IMHO overkill, you can just manually run the deploy on one host, wait for it to start working properly, and then upgrade the other.

"I don't think you would in prod, unless you were happy with it all running on one host."

1) create a VM, set it up with docker compose to auto start -- have it reference code on a network drive.

2) take a an image of 1)

3) create a load balancer, balancing that image. GCP load balancers/instance groups even a way to rollout images.

That is %80 of k8s right there (endpoint, scaling, containers, rollouts).

...am I the only one having ansible deploy compose across machines?

Oh man I’m currently looking into doing exactly that. Can you elaborate a little bit on how it works?

> In production I normally see people building production images and spinning up them up in their cloud with Terraform and whatever other infrastructure they need

I see a stark difference between infra management tool (like Terraform), and deploy tooling. Infra I prefer to describe "declarative", and Terraform helps a lot with that. Deplyments are "imperative" in nature to me: turn on maintenance page, remove cluster X, spin up cluster Y, run migrations, etc.

So I find Terraform a bad fit for deployment tooling (please explain if you disagree, I'd like to learn).

Why would you NOT choose this path? Local and prod are the same with this method, that’s a big advantage.

Docker Swarm is one option, but Docker Desktop has the ability to deploy from a Compose file directly to AWS. There are additional tools for other platforms. See https://aws.amazon.com/blogs/containers/deploy-applications-... for example.

We use docker compose in production, but have our own custom tooling based around it.

K8s was a little too complex and docker swarm a little too simple. Rolling our own solution was a last resort, but it worked out well in the end - more control, deeper understanding and no nasty surprises in behaviour while scaling etc.

I found that Hashicorp Nomad fit really well in between k8s and swarm for power to complexity. I ended up not being able to use it because they hadn't implemented the networking stuff that they eventually got, but it's pretty cool stuff.

I second this. Nomad is a nice middle. Like the post above I've built tools to fill that gap. What I don't understand sometimes is why there aren't many more orchestration tools in the ecosystem. Some tools emerge; work and then somewhere along the line get kubernetes feature envy and never quite get there.

My home setup is now k3s.

We deploy lots of small apps for internal usage, and we deploy those with docker compose.

If your app fits on a single VM, and you don't have k8s, this is incredibly straightforward.

It's even "easier" to just use kubernetes to deploy something - but that's if you already have kubernetes! As my org is adopting k8s we are favoring that over docker-compose where applicable, but there's a huge complexity gap between docker-compose and this isn't an overnight transition.

I guess you could view docker-compose as a much more accessible stopgap as your team learns how to run the more complex orchestration strategies, or a preferred solution if you have a small number of VMs you want to explicitly configure to run specific services on (where you don't need the scheduler etc).

The recent update to `docker-compose`, now called `docker compose`, has contexts that let you us the up command to deploy to services like Azure Container Instances or Elastic Container Services.

https://docs.docker.com/cloud/aci-compose-features/

https://docs.docker.com/cloud/ecs-compose-features/

I haven't (successfully) used ACI to deploy my compose app. I'm hoping somebody else on this thread might have.

I have played around with docker compose + ECS and it works great for simple use cases.

However, I prefer (at least for my needs) AWS CDK because it also supports pushing of the image to remote registry, and spawning other AWS resources, such as databases.

It's currently in preview, but https://docs.microsoft.com/en-us/azure/app-service/quickstar... doesn't look bad.

Same here, docker compose is a great little tool for local development. You can get your apps up and running with their respective elasticsearches and localstacks etc, then attach to process if needed to debug

For anything more serious, Terraform makes much more sense IMHO

Terraform and docker-compose can work well together. Terraform to describe the AWS infrastructure, and docker-compose to say what's running on a given VM.

Why would you bother using terraform+docker compose in AWS if you already get containerized solutions out-of-the-box with ECS, which is supported as first class citizen with CloudFormation/CDK?

May be he doesn't want to use CloudFormation? It's easier to switch to any other platform and why switching Cloudformation when everything is doing well with Terraform>

Exactly. My first choice will always be platform-agnostic tooling.

AWS elastic beanstalk supports docker compose files

For side projects I generally run docker-compose via docker-machine. One VM with all my images. Works great!

Reply on news.ycombinator.com