Awesome Docker Compose Examples
github.com
github.com
If anyone is looking for ready to go web app examples aimed at both development and production with Docker Compose, I maintain:
- https://github.com/nickjj/docker-flask-example
- https://github.com/nickjj/docker-rails-example
- https://github.com/nickjj/docker-django-example
- https://github.com/nickjj/docker-node-example
- https://github.com/nickjj/docker-phoenix-example
The examples use a combination of services for each tech stack such as web + worker + postgres + redis + esbuild + tailwind. The Rails example is set up for Hotwire and runs Action Cable as a dedicated service along with Sidekiq where as the Flask and Django examples use Celery as a worker. You can easily swap things out since the examples are starter projects that you can clone + rename (they all come with a rename script), you're meant to customize them to build your app on top of.
I can use Docker in a basic sense, but it is amazing to me how much black arts still exists for what has become a cornerstone of modern deployment. Lots of conflicting/dated advice about best practices. Unsure which advice is still required/applies to podman, etc.
What?! Unless you are using complicated networking, docker is simple. Dockerfiles are essentially just annoying-syntax shell scripts.
We have plenty of black arts in computing. Docker isn't one of them.
Maybe not black magic, but there are a lot of subtle optimizations for which there is much conflicting guidance.
[edit: added second clause]
If you run your container as a non-root user and create a user without setting a UID / GID it'll default to 1000:1000, so unless your Docker host's user isn't 1000:1000 then things work out of the box. A lot of this is general Linux knowledge around user / file permissions, not so much Docker.
> Should I be using Alpine or Debian?
Debian, no contest in my opinion.
> How do I handle loading certificates?
You can volume mount them or deal with SSL certificates in a way where Docker isn't involved such as running nginx on your Docker host directly or putting a load balancer in front of your app and handling SSL there.
> Should I use an override config for development?
You can use the same docker-compose.yml file in all environments and tweak things with environment variables. The compose file supports variable interpolation. Docker Compose profiles also let you control which services to run in each environment, it's even configurable by a single env variable.
> Do I still need to worry about Docker blowing a hole in iptables?
If you use -p 8000:8000, yes this will publish the port in a way where the outside world can access it. Likewise without Docker if you edit iptables to allow that port it will too. I wouldn't classify this as blowing a hole in iptables. This is "user configured application to make a port open to the world".
> Maybe not black magic, but there are a lot of subtle optimizations for which there is much conflicting guidance.
In the end my example apps address most of these issues for you. You're on your own with certificates since that varies on your deployment, but everything else is fully set up and ready to go and it protects yourself from blowing a hole in iptables since it only publishes the port to localhost by default, not 0.0.0.0. This would let nginx or another web server access it directly on your Docker host but no one else.
I've tried to use the docker compose command to limit the maximum number of CPUs used, I failed, even though it's possible: the support team managed to do it, but why did I fail? Because the doc and the design aren't great..
I maintain similar Django and Flask + compose stacks on behalf of the startup studio I work for so it’s fun to compare notes.
For our Django stack, for instance, we have also settled on Postgres, on celery+redis, and on whitenoise. black/flake8/isort also seem universally agreeable. We also throw in pyright and generally make extensive use of type hints.
For the front-end, we’re currently Create React App with TypeScript and Tailwind. I’d love for us to move away from CRA, so your use of esbuild is helpful to see. (I’d personally be happy using HTMX or Turbo/Stimulus but for the moment a JSON API backend with a React SPA front-end seems more comfortable for more of the CTOs who hop on board.)
We also supply some minimally opinionated glue at the API layer. On the back end we have a base View that provides a few helper methods for transmuting invalid Form instances to nice JSON replies that the TypeScript API invocation code works with gracefully. (We used to push DRF but have lots of feedback from older startups than ran with it and had regrets down the road.)
The key thing to understand is that all these things are shrouded in incomprehensible jargon and alien sounding names that make it incredibly intimidating to get started. But the fact is, it is a wide but shallow pool of jargon sitting on top of the same old computing fundamentals that have been around since the 1970s.
You will find if you know your fundamentals, then the jargon is far less difficult to overcome than it seems. You just need a bit of exposure and if you spend any amount of time playing with the tech it just starts to happen by osmosis. (If there are fundamentals you aren't solid on, treat it as an opportunity to bed that in - even these things are not generally super complex in the end).
I think that one sane way is specialization. Become an expert in writing nginx-ingress yamls in some big corporation. Another sane way is to throw away modern tech and stay with old tech. You can deploy perl cgi to OpenBSD just like you could 20 years away. Another way is to carefully select technologies which were proven for 7+ years. I'm trying to follow this way.
IIUC, if you write a web application in the MS world, you have Visual Studio, ASP.Net, and IIS.
There doesn't have to be piles of libraries.
But because I am also an old engineer who works with a lot of other old engineers we are lucky to have a shocking number of human-years’ accumulated experience not only selecting a pile of tools and frameworks but also sticking with them from zero to $BIGCO. Put another way, I’ve shot myself in the foot more times than I can count with these tools and I do it less often and in more esoteric ways these days.
I can’t claim the same degree of experience with all these tools, of course. Pyright is bleeding edge. Tailwind is still the new kid. Relative to (say) Python and Django or Flask, React is new too. Most of the pain and learnings come from these newer moving parts.
Is it “better”? That depends on the axes of evaluation. As a startup studio where the 70% case might be “SaaS that takes a back-office process held together today by Excel & email and makes it way better” the answer is: sometimes, unequivocally yes. Not just because of the tools and the potential velocity they can confer, but because of the kinds of teams we can build around them. Sometimes, plain old Django with nothing added is a clearly better choice.
In the end, every startup is its own snowflake. We try to select “starter stacks” that balance industry familiarity with our ability to offer meaningful operational perspective. We definitely don’t think of them as the final world.
This post by Tailwind’s author gives some more perspective: https://adamwathan.me/css-utility-classes-and-separation-of-...
Other than that I agree.
Go's ecosystem is also pretty fragmented so it would be difficult to create a solution that lots of folk would be happy with. I get the impression most folks who use Go want to pick everything themselves (not a bad thing, just something I noticed). Plus I'm only 1 person doing this in my free time with no income sources attached to these projects, I don't have the capacity.
With that said, most of the apps are very similar when it comes to the Docker bits. You could take any of the examples and replace XXX with Go. I'd suggest basing it off the Phoenix example because that one covers using a 2nd Docker build stage to create a release. With Go being able to compile a self-contained binary that pattern of using a build stage to copy that over would be similar.
For example, a load balancer running 1 level above your server or perhaps nginx running directly on your Docker host without Docker.
A while back I wrote up why I prefer running nginx outside of Docker at: https://nickjanetakis.com/blog/why-i-prefer-running-nginx-on...
I'd suggest watching and reading: https://nickjanetakis.com/blog/a-guide-for-running-rails-in-...
It's a full end to end walk through of the example Rails app and how it all works when it comes to Docker and Docker Compose. I just recorded it a month ago, it covers everything with up to date info vs the code in the repo.
Yep I've seen mrsk and even briefly chatted with DHH about it a day or 2 after he open sourced it.
I have mixed feelings:
On one hand I think it's fantastic Rails is starting to take on deployment officially. This is going to lead things to a better place in the long run. Generally speaking DHH has a really good track record for making things that feel good to use.
On the other hand, I think the project is trying to reinvent too many things that already exist and doesn't account for Docker Compose. For example, for literally the last 7 years I've been deploying any Dockerized web application with Docker Compose to 1 or more servers with about 15-20 lines of YAML using Ansible to set up the server and git to deploy the code. It doesn't matter if it's Rails, Flask, Django, Phoenix, Node, Go or whatever. It's all the same. Those 20 lines also include everything from taking a blank slate Debian / Ubuntu box to production ready, complete with self managed system updates, locking down SSH, iptables, various server configurations / optimizations, nginx, HTTPS, database backups, sane logging and everything else you'd expect.
I don't have anything ready yet but I've been slowly working on https://nickjanetakis.com/courses/deploy-to-production to assemble all of this into a course.
As for AWS ECS mentioned in the other reply -- it's great if you ever get it working, but it is a nightmare to learn and use.
Unfortunately, there's a huge gap between "single container" and "multiple, co-dependent containers". It's a far cry between services hosting a single container and ones offering to host entire stacks.
And I think you're exaggerating the "huge gap" there. Fly.io can run multiple docker containers for you. Their docs are great so it wouldn't take much effort to learn how to create an equivalent or better setup that covers everything Docker Compose does (and more).
How many RCEs are discovered per year in baseline Debian/Ubuntu? Seems far more likely that security holes are in the library/application code layered on-top of an image.
Harmful code pushed into large software repositories masquerade as something else is not novel and is starting to happen more and more.
Sure, the image is changing now only at intentional times (as opposed to just any restart), but you're still not getting an assurance that a "trusted" upstream isn't going to RCE you. Security updates & patches come out with such frequency that the number of windows of opportunity are still plentiful.
Better advice would be to use tags, but to vendor the images somewhere and point at the vendored copies. You still get the "doesn't change on random restart", you have an idea of what version you're on that the hash doesn't tell you, you don't have to deal with Docker hashes being the weird AF things they are (e.g., you side-step the confusion that library/foo@sha256:XXXXX == vendored/foo@sha256:YYYYY), and you're more respectful of upstream's bandwidth and you're not dependent on their reliability concerns.
(And frankly, given how poorly most of the industry handles security updates, I'd argue that sitting on a very loose tag might be more secure overall, because then security patches might be happening at random times, as opposed to not at all. The trade-off of the RCE is worth it, and there are ways for upstreams to mitigate the risk there, centrally. Most companies struggle with just vendoring, and few get to actually keeping that up-to-date, IME due to a lack of willingness to apply the necessary resources to the problem.)
It does, changes to the image could be pushed to "latest" or a specific tag, but the hash cannot change, once you've established that a specific hash is secure, it'll remain secure (or rather, as secure as you first established it as).
To be honest if you’re that concerned with dependency attacks like that then you should just be hosting your own image registry and building your images yourself, and then only being vulnerable to dependency attacks within the OS distributions and such.
You can use a image like golang:1.20.1-alpine3.17@sha256:48f336ef8366b9d6246293e3047259d0f614ee167db1869bdbc343d6e09aed8a and be able to both see the version (human-parseable)and the hash (machine-parseable)
Dependabot will update both the version and hash parts of the tag in a pull request. Pretty magical if you ask me. I haven't found a way for it to auto-apply yet but Renovate can do it if you want automatic updates.
So it will be even bigger pain in the keyboard to write them :(
It is a blend of v2, v3 and any newly added attributes (v3 some times supported less than v2), that is now called the compose spec.
Implementations are supposed to do feature detection based on what attributes are used in the file. Make sure your docker compose is new enough, otherwise it thinks absence of version field means v1!
After June 2023 Docker Compose v1 will officially no longer be supported. I probably won't switch right then and there to the new filename but that is at least one ball moving in the right direction to switch over to the new file.
The other problem is more of a human one. There's 7+ years worth of blog posts, videos and documentation referencing docker-compose.yml. Having all of those become invalid is a big price to pay when onboarding new folks to Docker. When learning something new, it stinks when you find conflicting or different information all over the place.
Long story short, I will switch to using it but it's going to take time.
If you tell me to run
docker run -v ${HOME}/stuff:/stuff alpine
Or just go with k8s if you need a more complex setup.
Also, its really nice to be able to use `docker context use SERVER_NAME` so I can switch contexts (servers) very easily.
Though, I'm still confused by your example of having to rsync /home/stuff to the server. If you use a named Docker volume, is the remote Docker container somehow using a volume you have located on your laptop? Wouldn't you still have to transfer the volume from laptop to server?
In my system all of the files get written to the volume from only three places:
* From the docker image through VOLUME (fresh volumes copy the data from the image on start)
* From a template container that writes config files.
* From the container itself, writing files as it runs.
When I run `docker build` on my laptop, this does copy files to the server (Docker designed build this way, and you have to set .dockerignore file to ignore files you dont want copied).
> If you tell me to run
>
> docker run -v ${HOME}/stuff:/stuff alpine
>
> It will mount /home/stuff on the server, not my own
> home directory on my laptop.
And then you note:
> What I don't do is create a directory someplace and manually edit files and mount them.
But if /stuff is photos, and another container, say, runs ingestion tools, or some other photo collection processing, you don't let it touch the same data volume?
Looking at your repo, I see your docker-compose volumes map e.g. data to data …
volumes:
- data:/data
For instance, mounting a volume that can be edited by other containers lets me insta-move large files or sets of files between steps of containers, by container a doing a move not copy from its work path to its destination path watched as an incoming path by container b.
In this extreme example I think probably a bind mount might make sense, especially if the files are already there. But the named volume would just be stored in /var/lib/docker/volumes/some-volume-name, so as long as that /var/lib has 10TB I don't see the problem.
I can use my sftp container [1] to be able to sftp directly into a volume, but I've not yet transferred 10TB with it :)
[1] https://github.com/EnigmaCurry/d.rymcg.tech/tree/master/sftp
… but this made me discover this:
docker volume create \
--driver local \
--opt type=cifs \
--opt device=//uxxxxx.your-server.de/backup \
--opt o=addr=uxxxxx.your-server.de,\
username=uxxxxxxx,\
password=*****,\
file_mode=0777,dir_mode=0777 \
--name cif-volume
Thank you for all the replies!
Theres two ways to mount a volume, one with a / and one without.
/some/directory:data
some-volume:data
The first mounts a directory
The second mounts a named volume.
I suggest doing the second so that it can be maintained directly by docker through `docker volume create|rm` or `docker compose up|down [-v]`.
> For instance, mounting a volume that can be edited by other containers lets me insta-move large files or sets of files between steps of containers,
As long as the containers are on the same docker host, many containers can mount the same volume.
Most of our App deployments are done with GHCR + SSH + Docker Compose with GitHub Actions on every commit [1]
[1] https://docs.servicestack.net/ssh-github-action-deployment
How so? For me it makes it easier, just have to backup one directory: /var/lib/docker/volumes
> you will need bind mount for external config files anyway
I create all my config files from templates, generated by a container, with the config entirely driven by environment variables, and this runs before my main container (via `depends_on`) it writes the config to a named volume. So there are no "external" config files, only "internal" ones.
* https://github.com/docker/awesome-compose
* https://github.com/PostHog/posthog
* https://github.com/growthbook/growthbook
* https://github.com/fleetdm/fleet
Supposedly, One can search github using `language:typescript filename:docker-compose.yml stars:>1000` but it's not working for me somehow.
Fanatics love docker, for sure. It's… fantastic!
I don't want the impedence mismatch of working locally with Docker and remotely with K8s.
That being said, K8s IS harder to setup initially.
Having to setup an NFS share locally to be able to properly mount data in a PV is suboptimal.
But part of me still makes me think about what I think might be the gold standard of self-hosted, cloud-like things, and that's Syncthing? As in, all of these things should try to figure out how to be like syncthing?
OMG, why?! It is even had a daylight savings!