Docker Compose Isn't Enough
blog.tealok.tech
blog.tealok.tech
There are simple solutions to each
1) you simple have a separate docker-compose file for a different environment, ie docker-compose.dev.yml for your dev server. in this file you simply define the parts that differ from the primary / prod compose file. that way it's a simple command. line variable that initiates a dev vs prod vs any other type. for details see https://docs.docker.com/compose/how-tos/multiple-compose-fil...
2) this is literally as simple as running a small bash script that creates a backup, tars and gzips the file and then uploads it to an S3 repository. Sure, maybe not out of the box, but generally pretty simple. Likewise i'd always have a script that does the inverse (downloads the latest backup, loads it into the db and restores the files).
Not exactly rocket science...
Sounds like your container has some kind of side-car that makes it directly addressable over Wireguard without needing to address the host IP. Does that mean you'd need to modify the docker-compose in some way before `docker-compose up`?
How do you know which port Paperless is using for HTTP? When you want to load up Paperless in a web browser, are you typing in a service name, or the container IP address? If it's a service name, how are you doing DNS? Do you have TLS?
Not necessary. You can access the deployed docker container without exposing any ports or having any reverse proxy (what you've likely thought about with sidecar, which is a k8s concept, not docker) or anything else by using the ipadress of the started container and the ports the started service used. This is usually only possible from localhost, but wireguard can be configured as what's essentially a bastion host and exit node, this would let connecting clients also address containers that were started on that server, without opening any ports.
You can technically also do that without wireguard even, as long as you configure the docker host to route relevant traffic into the docker ethernet and define the docker subnet as a static route that points to the docker host, but that's another story
Your explanation is interesting though. Would that actually work?
For services open to the public internet I just add a subdomain + a reverse proxy entry into an existing caddy instance and point to the docker IP.
I regularly try out new services or tools with docker-compose
For a lot of developers who have their primary focus set on writing code and documentation, trying to remember Docker or Kube on top of the rest of your stack is burdensome. Deployment is an end-user task. Obviously the development environment fits the target use-case already.
So please understand that there are lots of people who are building the things that go inside your containers who don't want, need, or see the value of these containers. For you they might offer value, but for someone like me they bring cognitive load and troubleshooting.
Too much abstraction can be just as bad as not enough. Like Akin's Laws of Spacecraft Design, but for software.
Eg
COMPOSE_FILE="compose.yaml:compose.prod.yaml" docker compose up
I'm not sure that his solution is really an improvement, however, because then you have to bake in the semantics for a fixed set of product types, and when that list gets extended, you have to bake-in a new one.
There are pre-existing paradigms for service brokers, message passing, etc., they all try to solve this problem, and they all share this same fault, they become in-extensible...
Which is the core programming tradeoff, flexibility vs robust constrained behavior.
On the extreme end of flexibility you have AI goop you barely understand, but arguably it is far more flexible than anything you ever wrote, even if you can't make any guarantees about what it does, on the other hand you have these declarative things that work fantastic if that's what you wanted, but probably not worth getting into computers for...
Some sort of "hey back this data up, it's important to my function standard would be nice, and not just for container apps.
Trying to figure out where your app hid its configuration and save files is a perennial user frustration.
It is easy to set the right storage behind /data/.
Just a convention, nothing more, smoothly working for the past "many" years.
This feels like quite a shallow reading. Being able to manage system-wide things (DNS, and a centralised Caddy, centralised database) from individual compose units in such a way that you're not going through a ton of setup each service. This much might just need Caddy to work better with other instances of itself but it isn't a solved problem.
I'll grant you that backing up volumes isn't hard.
It took me some time initially to figure things out and I had to learn some things. But now it's a breeze. Once I had reverse proxy and automatic certificate renewal in place it has been working ever since then without me having to do anything. Adding a new service like Immisch or Jellyfin takes me an hour or less. Which can be quicker, but I adjust every docker compose to my setup and make it more secure. E.g. I create a new non-root user for each service. Basically I have the whole setup figured out; I have notes and checklists for the new services I add. I don't need to figure out things anymore and in 95% of the cases things just work.
Updating existing services takes minutes: just increment the version in the compose file and rebuild.
For my setup, I use macvlan as opposed to the default docker bridge network. So, the `ports: - "8000:5000"` docker compose config is being ignored. Instead, each docker container gets its own unique IP and MAC address I assign to it. It can then use any port it wants. Thanks to this, on my home network, I can access any container I want by IP. And any container can access any other container. I then add some restrictions for security purposes on the host machine using nftables.
For reverse proxy, I use Nginx Proxy Manager which is a simplified GUI around Nginx. I'm slowly moving to just using Nginx.
For the database, I run multiple instances of Postgres and MySQL and I don't see any issues. They take an insignificant amount of resources compared to the applications. If the application is not in use, its database instance is not being used as well.
Also aside, "docker compose" (V2) is different from "docker-compose" (V1) [0], it was rewritten and integrated into docker as a plugin. It should still be able to handle most old compose files, but there were some changes.
There's Docker swarm mode for that. It supports clustering too.
It's nuts how people look at a developer tool designed to quickly launch a preconfigured set of containers and think it's reasonable to use it to launch production services.
It's even more baffling how anyone looks at a container orchestration tool and complains it doesn't backup the database they just rolled out.
> In that context many people find k8s or other PaaS to be too heavyweight so Docker Compose is pretty popular.
...and proceed to put pressure to shitify it by arguing it should do database backups, something that even Kubernetes stays clear from.
The blogger doesn't even seem to have done any research whatsoever on reverse proxies. If he would have done so, in the very least he would have eventually stumbled upon Traefik which in Docker solves absolutely everything he's complaining about. He would also have researchdd what it means to support TLS and how this is not a container orchestration responsibility.
Quite bluntly, this blog post reads as if it was written by someone who researched nothing on the topic and decided instead to jump to
I'm curious how that last sentence was going to end.
Let's say I agree with you and that TLS termination is not a container orchestration responsibility. Where does the responsibility of container orchestration start and TLS termination end? Many applications need to create URLs that point to themselves so they have to have a notion of the domain they are being served under. There has to be a mapping between whatever load-balancer or reverse proxy you're using and the internal address of the application container. You'll likely need service discovery inside the orchestration system, so you could put TLS termination inside it as well and leverage the same mechanisms for routing traffic. It seems like any distinction you make is going to be arbitrary and basically boil-down to "no true container orchestration system should care about..."
In the end we all build systems to do things to make people's lives better. I happen to think that separating out backups and managing ports as an exercise for the deployment team raises the barrier to people that could be hosting their own services.
I could be totally wrong. This may be a terrible idea. But I think it'll be interesting to try.
> If he would have done so, in the very least he would have eventually stumbled upon Traefik which in Docker solves absolutely everything he's complaining about
I'm aware of Traefik, I ran it for a little while in a home lab Kubernetes cluster, and later on a stack of Odroids using k3s. This was years ago, so it may have changed a lot since then, but it seemed at the time that I needed an advanced degree in container orchestration studies to properly configure it. It felt like Kubernetes was designed to solve problems you only get above 100 nodes, then k3s tried to bang that into a shape small enough to fit in a home lab, but couldn't reduce the cognitive load on the operator because it was using the same conceptual primitives and APIs. Traefik, reasonably, can't hide that level of complexity, and so was extremely hard to configure.
I'm impressed at both what Kubernetes and k3s have done. I think no home lab should run it unless you have an express goal to learn how to run Kubernetes. If Traefik is as it was years ago, deeply tied to that level of complexity, then I think small deployments can do better. Maybe Caddy is a superior solution, but I haven't tried to deploy it myself.
It isn't. It's not a problem, either. That's my point: your comments were in the "not even wrong" field.
> (...) It seems like any distinction you make is going to be arbitrary and basically boil-down to "no true container orchestration system should care about..."
No. My point is that you should invest some time into learning the basics of deploying a service, review your requirements, and them take a moment to realize that they are all solved problems, specially in containerized applications.
> I'm aware of Traefik, I ran it for a little while in a home lab Kubernetes (...)
I recommend you read up on Traefik. None of your scenarios you mentioned are relevant to the discussion.
The whole point of bringing up Traefik is that it's main selling point is that it provides support fo route configuration through container tags. It's the flagship feature of Traefik. That's the main reason why people use it.
Your non sequitur on Traefik and Kubernetes also suggests you're talking about things that haven't really clicked with you. Traefik can indeed be used as an ingress controller in Kubernetes, but once deployed you do not interact with it. You just define Kubernetes services, and that's it. You do interact directly with Traefik if you use it as an ingress controller in Docker swarm mode or even docker-compose, which makes your remark even more baffling.
> I'm impressed at both what Kubernetes and k3s have done. (...) If Traefik is as it was years ago,(...)
Kubernetes represents the interface, as well as the reference implementation. k3s is just another Kubernetes distribution. Traefik is a reverse proxy/load balancer used as an ingress controller in container orchestration systems such as Kubernetes or Docker swarm. The "level of complexity" is tagging a container.
Frankly, your comment sounds like you tried to play buzzword bingo without having a clue whether the buzzwords would fit together. If anything, you just validated my previous comment.
My advise: invest some time reading on the topic to go through the basics before you feel you need to write a blog post about it.
i've run production instances with 2PB of data being scraped per month by 100+ CPUs and 256GB of RAM using docker compose. some of the machines (smaller instances) have run flawlessly with zero reboots for years on end. both on cloud and on-prem.
i'd say its 100% production ready.
That's fine. Some people also manually launch containers individually with Docker as their production system.
They'd be wasting their time and making their lives needlessly harder, though.
They already have tools that do all the legwork for them. Why not do the research and opt to force a square peg into a round hole?
Research Docker swarm mode, reverse proxies like Traefik, and Let's Encrypt. Kubernetes can be even easier with implementations like microk8s, which are a snap away from any Ubuntu installation. Look at what you're doing and figure out the right tool for the job. Don't just whine about how you're doing something wrong and it's the tool that needs to change to fix your own mistakes.
Using Kubernetes where it's not needed is just that - wasting your time and making your life harder.
Before you say that I just need to research more: I know Docker swarm mode, I run my personal server on Kubernetes using Traefik and Let's Encrypt, I professionally work with Kubernetes (both as an admin and working on Kubernetes security, which is tough to get right), most services in my dayjob run on Kubernetes, and I was the person who introduced CI/CD pipelines there some years ago.
I still claim that there are production usecases that are better served by docker-compose.
I think this is a very ignorant and misguided take.
Kubernetes is a container orchestration system, just like Docker swarm mode or even Docker compose. If you need to deploy sets of containerized apps into your own VMs, you can pick up any Kubernetes implementation. You don't even need Helm or anything. Plain old kustomize scripts will do. Some aren't even longer than a docker-compose.yml.
More to the point, Kubernetes is an interface. One that you can use in local deployment and in major cloud providers.
You should really check your notes because your comments contrast with the realities of actually running a service.
> Before you say that I just need to research more: (...)
All your appeals to authority are falsified by your claims.
I, on the other hand, actually use Kubernetes both professionally and in personal projects, as well as Docker swarm mode, and can tell you I'm no uncertain terms that none of your points have any traction in reality.
> I still claim that there are production usecases that are better served by docker-compose.
I'm sorry, but your comments simply sound deeply uninformed and misguided.
I mean, it makes absolutely no sense to comment on using docker compose in production when Docker swarm mode is far more capable and Docker swarm mode stacks already share most of the schema with docker compose. You literally have virtually nothing to do to adapt a docker-compose script to launch a stack.
Unbelievable.
That's perfectly fine. That's not the reason why you are better off running your apps in Kubernetes though.
You are better off running your apps in Kubernetes because it handles everything you ever need to effortlessly run containerized apps. You don't even need to install tooling or deployment tools or anything at all. You can have a single kustomize script that defines your services and deployments, have in place an ingress and Let's Encrypt, and you're done. You don't need to bother with anything else. Just run kubectl apply and go grab a coffee.
Is this something no one would ever need?
Hey that is not true, at least last time I tried I spent considerable time trying to integrate a bare metal load balancer from a 9 stars github repo plugin because apparently exposing the port is not recommended. Also having the master and the node in the same server can be problematic because by design it shouldnt be like.
One last point, the runtime ram and cpu overhead is far from minimal.
You sound like you tried to put together an ad-hoc ingress controller without knowing what an ingress controller is and why abusing NodePort is a mistake in the context of a cluster.
You can check this out to go through the basics.
https://kubernetes.io/docs/concepts/services-networking/ingr...
> Also having the master and the node in the same server can be problematic because by design it shouldnt be like.
You should look at the problem to see if an approach makes sense. If you're using vanilla Kubernetes on a one-box deployment then you're abusing a large-scale high-scalable cluster management system by shoving it into a single box. It can work, but most of the resources will be wasted on managing your cluster of one node.
There are plenty of Kubernetes distributions that are designed to handle well small clusters, and even one-box systems. Minikube, microk8s, and even k3s come to mind. I'm partial towards microk8s because installing it is just a matter of installing a single package from the distro's official repository.
So it's 100% production ready... for you.
That plus a decent backup system would work for a small array of servers with fail-safes. Though I would die inside if a production user-focused app under any level of proper load was set up like that.
We needed to do inference on remote machines stationed at random points across North America.... the machines had work to do depending on external factors that were rigidly scheduled. really easy to make upgrades, and the machines were _very_ beefy so docker-compose gave us everything we needed.... and since we shipped the data off to a central point regularly (and the data had a short shelf life of usability) we could wipe the machines and do it all again with almost no consequences.
I needed to coordinate a few small services and it did the trick beautifully
What's vital is being able to roll back a recent update, recover from backups, and deploy from scratch, all quickly. Those are usually (not always) far easier to achieve than ultra-high-uptime architecture (which also needs those things, but makes them all more complicated) and can be simple enough that they can be operated purely over ssh with a handful of ordinary shell commands documented in runbooks, but skipping them is how you cheap out in a bad way and end up with a system down for multiple days, or one that you're afraid to modify or update.
There is already a third version of compose.
far more useful for development IMO, but when push comes to shove and you need a few things running together on a machine in the middle of nowhere with limited/unreliable internet access.... you can't do much better... a few volume mounts for durability and you're really in decent shape.
logs, networking, system health etc etc.... just all a few docker and docker-compose commands away
Tons of sites are launched manually and have been for years in production.
Docker compose is a huge step up from that, so why wouldn't it be?
https://docs.docker.com/compose/intro/features-uses/
Sounds like dev, and test tool with production aspirations that isn't actually there yet to me.
I'm quite skeptical that adding a layer of abstraction and switching to TOML instead of YAML will suddenly enable those scared away by compose to start self-hosting, but kubernetes and docker swarm were never in the cards.
> I'm quite skeptical that adding a layer of abstraction and switching to TOML instead of YAML will suddenly enable those scared away by compose to start self-hosting, but kubernetes and docker swarm were never in the cards.
Yes, this is an excellent point. I did not articulate it well anywhere, but the goal is for users to have something more like Sandstorm, with a UI to install things. The TOML is for application developers, not end users. It'll either go in a separate database or, ideally, in the source code of the applications to be installed similar to a Dockerfile. I haven't started yet, but eventually we need to work with application developers to support things they want and to make it easier to treat Tealok as the "easy option" rather than docker compose.
The trickiest thing doing it this late in the game is going to be that docker compose has truly become the standard at this point. I self-host a ton of different apps and I almost never have to write my own docker compose file because there's always one provided for me. At this point even if your file format is objectively better for the purpose, it's going to be hard to overcome the inertia.
What do you use to manage all those compose files? Do you have off-site backups? I'm constantly reading and re-writing docker-compose and bash scripting everything to fit in with the rest of my infrastructure it'd be good to hear about someone with a better way.
I periodically run Renovate [0], which submits PRs against the infrastructure repo on my local Forgejo to update all my applications. I have a script in the repo which pulls the git changes onto the server and pulls and restarts the updated apps.
Data is all stored in volumes that are mapped to subfolders in a ~/data directory. Each application has a Borgmatic [1] config that tells Borgmatic which folder to back up for that app and tells it to stop the compose file before backup and resume it afterwards. They all go to the same BorgBase repository, but I give each app its own config (with its own retention/consistency prefix) because I don't want to have network-wide downtime during backups.
At the moment the backup command is run by me by hand, with BorgBase configured to send me emails if I forget to do it for a week. Eventually that will be a cron job, but for now it takes less time to just do it myself, and I don't change my data often enough for a week of lost work to hurt much.
All the applications bind to ports which are firewalled, with Caddy and Pihole being the only applications that run on exposed ports (53, 80, 443). Caddy has a wildcard DNS cert from LetsEncrypt for HTTPS and directs traffic from a bunch of local domain names to the correct applications. I just use Pihole to define my local DNS names (custom.list, which is where Pihole keeps the local DNS definitions, is a volume that's committed to the repo).
The problem that the author refuses to accept is that deployment is inherently complex. It does require configuration of a lot of things because it supports every use case. So you either have a generic complex tool that everyone can use (like Docker Compose or Kube) or you have some specific tool that only works for a tiny subset of all users that is simpler that satisfies your use case.
Note that I’m not saying Docker Compose is perfect. The syntax is a bit arcane it’s complex to understand etc. But removing the complexity by removing configuration options is not the solution here. Instead the author should focus on different approaches and tools to manage the same existing level of abstraction.
And for what it’s worth, that’s essentially what Helm is for kube - a way to manage and hide away the complexity of kube manifests (but still use those manifests under the hood). But frankly, docker compose doesn’t need a helm. Because docker compose, as you point out, has value not as a deployment tool, but as a single file that developers can manage and spin up on their local machines in a manageable way that doesn’t have the author fighting YAML all day.
I would say if the author was actually interested in solving this problem in a productive way they should first try to see if docker itself is amenable to altering their constructs to provide optional higher abstractions over common concepts via the compose interface natively. If the source tools roll out those abstractions everyone will get them and adopt them.
docker-compose is a lot of things to a lot of people. When it was created I doubt anyone realized it would eventually be the de facto standard for deploying to homelabs. It's an amazing tool, but it could be better for that specific use. I don't think that segment is important enough to the team that maintains it to warrant the change you're suggesting.
Most places (though not all) that I've seen using docker or docker-compose are throwing them in systemd units anyway.
With Kubernetes, it starts new container on new machine, and you don't get paged.
You could write scripts to do failover, but that takes works and will be buggy.
Either way this type of argument just comes down to "should I cluster or not" but to think out loud for a bit that's just basic HA planning: Simplest solution is keepalived/etc for stateful services, standard load balancing for stateless. Don't want load balanced services running all the time? Socket activation. Don't have a separate machine? Auto-restart the service and you can't cluster anyway. The only thing you'd really have to script is migrating application data over if you're not already using a shared storage solution, but I'm not sure there's any easier solutions in Kubernetes
Also even in separate components it's not really "all of that". Assume an example setup where the application is a container with a volume on an NFS share for state: on a given node we'd need to install podman and keepalived, a .container and .volume file, and the keepalived conf. An average keepalived conf file is probably 20ish lines long (including notification settings for failover, so drop like 8 lines if you don't care about that or monitor externally) and looking at an application I have deployed a similar .container file is 24 lines (including whitespace and normal systemd unit file boilerplate) and the NFS .volume file is 5 lines. So ballpark 50 lines of config, if you or others wanted to compare configuration complexity.
Also, fun fact, you could still even use that Kubernetes manifest. Podman accepts .kube files to manage resources against itself or a Kubernetes cluster; I've been recommending it as sort of a middle ground between going all in on k8s versus transitioning slowly from a non-clustered deployment.
Now it's 5 years+ running, all fine. No major hassle (crossing fingers!), no configuration hell. Never touched k8s, and not going to in the near future.
Maybe it just works and is sufficient for my case.
I don't think it's a good idea to spend time and resources learning complexity you never need. You are probably going to say that's not true as your cloud hoster has this all set up, but yeah, that's where the making profits comes in. We save millions/year on not cloud hosting. And managing your own k8s cluster apparently is quite hard (at least that's what even the fanboiz here say).
Starting today, I would use Go probably instead of Perl, but I would do the same. It's much simpler than kubernetes. But sure; for our goals; we like simplicity and don't need resumes to drive.
> Tealok is a runtime we’re building for running containers.
If you have one machine and docker-compose is falling short, really, just write a Python script with the official docker Python package, you'll be fine.
I found Watchtower, but polling just struck me as the Wrong Answer. Both too much overhead to keep pinging for the latest builds, and too slow to actually download the latest build. So I took some hints from Watchtower as to what I needed to do (mount the docker sock as a volume) and wrote a tiny Go server that, when pinged with a shared secret, would cause it to run `docker compose up -d --pull always`.
Probably took me an hour.
Then I added the ability to purge images before each update, because my tiny VM kept running out of disk space in the Docker partition. Oops. Scripts FTW.
I was already using the suggestion in the article about having a single reverse proxy server to redirect different paths (and different domains) to different servers hosted in the Compose file. Seemed like the obvious answer.
And I've configured k8s for my day job, so I could be using that. But I'm using Compose because I know how much of a pain k8s can be, especially around upgrades, where it has a habit of deprecating older versions of various interfaces. I'll put in the work for someone who's paying me to do it, but I'd rather work on my side project, not on configuring k8s.
Is the code for this public? I had the same desire when setting up the server for my personal project but as you mentioned I eventually decided it was OK to just poll every two minutes and I would rather work on the project than on configuring docker. What I would like though is cleaning up the older images that are no longer needed.
Some people would rather write 1000 lines of yaml than 100 lines of python, and I really don't understand why.
It’s often easier for developers to write a script using standardized tooling than dig into deep configuration docs for a complex application you’re not familiar with, but that’s where the benefits stop. Configuring built-in functionality makes sense for the same reason using an existing framework/os/etc authentication system makes sense. It often seems like way more effort to learn how to use it than rolling your own simple system, but most of that complexity deals with edge cases you haven’t been bitten by yet. Your implementation doesn’t need to get very big or last very long before the tail of your pipeline logic gets unnecessarily long. Those features don’t exist merely because some people are scared of code.
Additionally, if you’re just telling the application to do something it already does, writing code using general purpose tools will almost certainly be more verbose. Even if not, 10 to 1 is fantastically hyperbolic. And unless you write code for a living— and many dev ops people do not— defining a bunch of boolean and string values to control heavily tested, built-in application functionality (that you should understand anyway before making a production deployment) requires way less mental overhead than writing secure, reliable, production-safe pipeline code that will then need to be debugged and maintained as a separate application when anything it touches gets changed. Updating configuration options is much simpler than figuring out how to deal with application updates in code, and the process is probably documented in release notes so you’re much more likely to realize you need to change something before stuff breaks.
If you can't code, then yeah, I bet config is easier. But as a person who codes every day, I much prefer something that I can interact with, test, debug, type check, and lint _before_ I push to prod (or push anywhere, for that matter).
I’ve been writing code for about 30 years, worked in systems administration for a decade, and worked as a back-end web developer full time for over a decade. I’ve dealt with enough code as business logic, code as config, and parameter configuring to understand that errors stem from either carelessness and/or complexity, which is often a result of bad architecture or interface/api design. The more complex something is, the less careless someone has to be to screw something up. Adding logic to a config unambiguously adds complexity. If you haven’t lost hours of your life to complex debugging only to find it was the wrong operator in an if statement or the like, you’re not a very experienced developer. That’s that. You can have all the machismo you want about your coding skills but those are unambiguous facts.
Developers doing their own ops and systems work have perennially been a liability for system-level architecture, stability, and security from multiuser computing’s inception. That’s why mature organizations have subject matter experts that take care of those things who know that a flat file of parameters and good docs is a whole lot more palatable when the pager goes off at 2am than a 100 line script the mighty brain genius developer made because they wanted to do it “the easy way that verysmart people know how to do” rather than learning how the config system worked.
Nitpicking: You can't really make up for not having a zoom lens as your field of view will stay the same. Hitchcock's famous "dolly zoom" demonstrates that quite nicely.
… aand there is even Ansible playbook for that! Maybe overkill.
If they can pull it off more power to them, but I'm skeptical that what the non-technical self-hoster needs is a TOML DSL that abstracts away ports, volumes, and even database implementations behind a magic text file. At some point you just have to bite the bullet and admit that your audience needs a GUI (like Sandstorm [1]).
> I'm skeptical that what the non-technical self-hoster needs is a TOML DSL that abstracts away ports
I fully agree, the end user would not be writing TOML DSL files. The end user would get something much closer to an app store, or what Sandstorm did, with one (or a few) click installs. The TOML DSL would be written by developers familiar with the application and stored either in a separate database, or ideally in the applications source control like a Dockerfile.
Oh hai.
Honestly I'm not sure I'm a reliable source for why we failed. It's tempting to convince myself of convenient excuses.
But I really don't think the problem was with the idea. We actually had a lot of user excitement around the product. I think we screwed up the business strategy. We were too eager to generate revenue too early on, and that led us to focus efforts in the wrong areas, away from the things that would have been best for long-term growth. And we were totally clueless about enterprise sales, but didn't realize how clueless we were until it was too late (a classic blunder). Investors really don't like it when you say you're going to try for revenue and then you don't, so we were pretty much dead at that point.
It sounds like you have no criticism of the technical approach, then, but rather just the business mechanics? That's eye-opening, given how much has changed in self-hosted deployment since Sandstorm started. If you started something similar today, ignoring business needs, would you build something technically similar?
You should probably take me with a grain of salt: of course, I, the technical architect of Sandstorm, still think the technical architecture is great. ;) Whereas I never claimed to be good at business so I'm not adverse to reporting I am bad at business.
But yeah I still think it's the right architecture. Still believe in what's written here: https://sandstorm.io/how-it-works
But I do think there is a lot of work needed to get to a baseline of functionality that people will use. Bootstrapping a new platform is hard and needs a long runway. Could you find investors willing to back it? I don't know. I hated fundraising, though.
Of course, there are many lower-level details I would change, like:
* Consider using V8 isolates instead of containers. Sandstorm had a big problem with slow cold starts and high resource usage which V8 isolates could do much better with. It would, of course, make porting existing apps much harder, but ports to Sandstorm were always pretty janky... maybe would have been better to focus on building new apps that target Sandstorm from the start.
* Should never have used Mongo to store platform metadata... should have been SQLite!
* The shell UI should have been an app itself.
* We should never have built Blackrock (the "scalable" version of Sandstorm that was the basis for the Oasis hosting service). Or at least, it should have come much later. Should have focused instead on making it really easy to deploy to many different VPSes and federate per-user instances.
It’s been super frustrating in the past to be stuck on expensive PaaS vendors, but then rolling a deployment solution from scratch ended up with us trying to stitch together GitHub actions.
Been just using canine for my own projects and I’ve been able to host 4 rails & 1 Rust app pretty seamlessly with it on a $24/month digital ocean server. Would love any feedback!
The magic with k8s is being able to easily host third party packages via helm. I often find for anything I want to run in production, I usually want something like sentry, grafana, etc, and those items can get expensive if you try to buy hosted versions. Kubernetes makes it trivial to just host it yourself.
In terms of performance, I’ve gotten reasonable performance out of a $24 kubernetes server, which is 2vCPU + 4GB memory. These days, DigitalOcean and linode don’t charge a markup on their managed kubernetes at all, above their regular VPS pricing.
Heztner is much cheaper than both those options though, and they don’t have a managed kubernetes product
https://harbormaster.readthedocs.io
It's a very thin layer over Compose, basically just auto-pulls git repos and restarts containers if they have any new changes.
I'm still not clear from the website what Tealok actually is in terms of a product offering, but I appreciate that the blog post was legit and not just SEO spam.
It has a very brief introduction to Docker and Docker Compose, but then it mostly has a lot of FUD that's designed to scare people off of Docker Compose but is not remotely convincing to someone who's actually worked with it. The way they frame that pihole example ("Whew! That’s a lot of stuff!") is just silly.
Looking at their website, I think they started out trying to make self-hosting really easy for a barely-technical user [0], which accounts for their attitude towards docker-compose, but it also means that the post was pretty devoid of useful information for someone who actually wants to self-host with the tech that's already ubiquitous today.
Yeah, you're probably right. Originally that line was in there when I had a breakdown of what each line in the docker-compose was doing. My editor thought that was unnecessary - it's unlikely people reading the post would need that kind of breakdown. So I rewrote parts to assume more baseline knowledge. I should have noticed that line and taken it out.
You're right about what we're trying to do, and I agree that the post doesn't really help someone be successful today deploying things. The post is more meant to gauge whether or not I'm alone in having pain deploying a couple dozen services with docker compose on a single box.
I want more people to have the power to host their own services. I think we can do that, but we have to figure out the right thing to build to do it.
Looks like we may share a company name with some grifters. That's what we get for making our name an obscure mythological reference.
So what is the real business opportunity here?
It says docker compose is at the "wrong-level of abstraction", but I kept feeling the author was instead expecting docker compose to solve different problems that it was ever meant to solve.
In fact, they seem to be expecting a highly-opinionated, high-level interface which solves problems that I don't think anyone using docker compose in prod should even be worried about.
A lot of concerns seem to be around avoiding spinning up duplicate instances of reverse proxies, databases, and caches. First of all, why is this a concern? Idle threads have basically no impact on a system, so this generally isn't a concern. This is a nuanced and context-dependent issue, but generally it won't even make the list of top-100 performance bottlenecks for most applications. Yet the article takes for granted that the benefits of solving this problem outweigh the many cons of coupling them together.
Even if you wanted to enforce your applications sharing a postgres instance under the hood, why would you want that to be black-magic performed by the container orchestrator?
Other stuff like DB backups just don't seem like issues docker compose users have. If you need to orchestrate across multiple nodes in order to meet your SLOs, then don't use docker compose.
Finally, it seems like the actual solution is significantly under-discussed. I both have tons of questions about how it's supposed to work, and I see lots of shortcomings with the parts that I do understand.
Beyond the specific issues I see, the fundamental attitude seems to be "force everyone to architect their applications in a very specific way, and don't even try to support any use cases which fall outside of it". You need a damn good reason to be that opinionated about these sorts of things, and it by definition will only work well in specific contexts. I'd be interested to read an article which tried to articulate why such an opinionated API would improve SDLC-considerations over docker-compose, but I don't think that's the article I just read.
> First of all, why is this a concern? Idle threads have basically no impact on a system, so this generally isn't a concern
Idle threads have very low impact on CPU utilization, probably, if the application is well-behaved (and I expect most databases and caching layers to be well-behaved in this way). The application itself, however, will need memory and the way containers are built prevents the usual de-deplication of system libraries.
> but generally it won't even make the list of top-100 performance bottlenecks for most applications
True, but it makes the short list of "how much stuff can I run on a $100 computer", and it's one of the relatively few concerns an application operator has when they are not the application developer.
> Even if you wanted to enforce your applications sharing a postgres instance under the hood, why would you want that to be black-magic performed by the container orchestrator?
To make self-hosting much simpler. If the container orchestrator doesn't do it, what do you think should do it?
> Other stuff like DB backups just don't seem like issues docker compose users have. If you need to orchestrate across multiple nodes in order to meet your SLOs, then don't use docker compose.
The DB backups are meant for disaster recovery rather than supporting multiple nodes. I guess that's multiple nodes through time... But, yeah, I agree, docker-compose is not a good fit.
> Finally, it seems like the actual solution is significantly under-discussed. I both have tons of questions about how it's supposed to work, and I see lots of shortcomings with the parts that I do understand.
Yeah, agreed, I'll be writing other things to discuss what I think the correct solution should be. I'm curious to find out if other people have existing solutions to the problems I outlined. If it's a solved problem and I just don't know about it, that'd be better.
> I'd be interested to read an article which tried to articulate why such an opinionated API would improve SDLC-considerations over docker-compose, but I don't think that's the article I just read.
It is not, and you're right, it needs discussion.
I'll have to look in to that.
And Swarm is barely more complicated than compose anyway. Don't know why people who are claiming they use compose in production just don't learn the few extra steps to do what Docker recommends.
I just took a quick look into it and (classic) Swarm was deprecated and ... replaced by Swarm (mode).
It basically looks for a custom external hostname property which is used to generate the haproxy configuration and issue SSL certs.
For ports I personally prefer to just use a consistent container port as much as possible (eg: 80) - I'm sticking a reverse proxy in front of everything anyway so no need for unique ports bound to the host that I won't remember.
Data I basically just bind mount into a structure that I can easily backup from the host. I've also managed to do some major postgres upgrades without any issue (https://github.com/mnahkies/shoe-string-server/blob/master/d...)
I wouldn't call this approach production grade - I'm very much in the use managed Kubernetes/SQL camp there, but it's been working great for my personal needs
Although we're currently migrating to Kamal [2] to take advantage of its nicer remote setup, management and monitoring features.
[1] https://docs.servicestack.net/ssh-docker-compose-deploment
I like the ideas mentioned though! Docker Compose is pretty good, but something more declarative would definitely be a plus. Would love to see where the project goes (and maybe will borrow some things for the future versions of mine :-)
It was written as a makefile that wrapped docker compose commands, but that's just because it was the easiest to mvp.
I considered it a "framework for running a large collection of containers locally".
I solved the ports issue by using env vars like a switchboard, with an iPortService=<inside container port> and oPortService=<outside network port unique to network-layer>
Then i used two directories, one for a data layer that ran datastores, and one for app, for apps. You controlled the layers as groups.
You could operate the layers separately, initialize the dbs, and also reset their storage state definitely.
Then I had to solve the "docker ps" output problem, since its unreadable. So, i just split it to 3 commands: state,ports, & running. It outputs what you expect.
I should re-write it into better code and open source it. It was fun to build.
I suspect I probably felt like that at one time but it seems fine now after repeated use.
Kubernetes makes this so easy once you get past the YAML and abstractions, so I'll probably FINALLY move this stack over to it. Long overdue.
And this is coming from someone who for the longest time recommended that developers new to containers start with compose before considering Kubernetes!
Incredibly easy to setup and manage for usual small scale deployments. I use it as one node swarms, and I have setup - backups - automatic https certificates setup and renewal - automatic upgrades to new images - easy setup of persistence of data on the server
I'm very surprised it is not more popular, with quite some people trying to replicate swarm's features with docker compose, but often in harder to maintain setups.
https://dockerswarm.rocks/swarm-or-kubernetes/ says "it's not sensible to build a new product using Docker Swarm Mode"
I'm not expecting to convince anyone but wanted to share an alternative approach(only applicable to certain setup)
We are moving our EKS workload over to ECS over the next year. I expect needing to down size my team because of it.
One thing K8s is not is cheap. That shit takes a well oiled team or a couple of hot shots so do right. We probably did a lot of what makes it expensive to ourselves by not switching to managed add-ons sooner and never evolving the apps that run in the cluster. I've only been lead for about 5 months now, but I'm finally able to start making significant progress on the necessary evolution that I've been trying to make happen for 2 years before my promotion. The enterprise is a big ship. Takes time to turn. Thanks for reading what turned into a rambling vent session.
Imo Docker Compose absolutely should not handle databases and web servers. It should orchestrate containers. It's up to the user what kind of containers should be orchestrated.
Also the author somehow never got the idea to run the reverse proxy in a container? You don't need to do any port mapping then.
TL;DR: Author believes docker compose is too complicated, and has a grudge against YAML for some reason. Author proposes an alternative configuration syntax that hides implementation details behind named templates.
So despite what the author wants us to believe, this isn't ACTUALLY a replacement for docker compose. This is yet another "easy-to-use" container orchestration product where there's extra upstream development between you and the docker image. Docker compose can run anything you can stuff into a container. This cannot, without some additional development. That may be value in that, but I'm not sure blasting docker compose as old and busted right out of the starting gate is a terrific marketing strategy.
"Docker-compose is a tool for working with Docker containers. It solves very real problems with deploying complex applications. By itself it is not enough to make self-hosting applications simple enough for the mass-market."
I'm using Synology NAS which has a nice GUI manager for Docker-Compose. So I put all the data volumes on btrfs, and I snapshot them every day. Then I sync snapshots to Wasabi via duplicacy.
It works amazingly well. No problems whatsoever. I guess having a nice front-end that shows the overall overview would be nice, but I can live without it.
I like to name some such attempts: - .NET Aspire - Avito Plato (home grown PaaS of ¨russian Amazon") - infrastructure layer of our ZIIoT (MES platform on top of k8s)
I wrote a backup tool for compose/swarm based setups based on restic but ended up having to reimplementing half of traefik because there was no more straightforward API and compose really is just a thin layer over docker.
The obvious example was certificate management. Cert-manager alone is almost worth setting up K8s if you have many ingresses/fqdns.
Article wasn't convincing enough about docker compose's shortcomings. The linked github's README even less convincing. But to each his own.
There's some (namespaces) knob to avoid that, but the lack of nod to it makes me worried.
OTOH containers as security boundary is iffy. But I still like them to not be root in case of compromise
The solution to the complexity of applications that are distributed as collections of interdependent containers is to put all of the different pieces inside a single container. This is equivalent to what people did before docker existed, and it's still perfectly viable today.
There's no need to over-complicate things.
Podman for "like Docker, but made by Linux people". It supports both application containers and system containers (which have systemd as PID 1).
Accidental complexity, accidental complexity, accidental complexity and when we use a tool that is designed for problem A for a problem B it is not sufficiently dealing with complexity. This is why we need a new tool.
¯\_(ツ)_/¯
Anything more complex deserves a full container orchestration platform.
I'm not convinced anyone below google needs kubernetes, but containers for quickly setting up and running something on my machine are a blessing.
When you have a open-source application with five or six moving parts (monitors, backend, a database, etc.) being able to deploy it to a VPS with a single docker compose and have them all containers act on an internal dockerized network without conflicting ports, etc. is a GOD SEND.
For instance, you want to run one program that was written for Python 3.y, but also another program written for Python 3.z. You might be able to just install 3.z and have them both work, but it's not guaranteed. Worse, your OS version only comes with version 3.x and upgrading is painful. With docker containers, you can just containerize each application with its own Python version and have a consistent environment that you can run on lots of different machines (even on different OSes).
They're also a lot more convenient than having to go through the arcane and non-standard installation procedures that a lot of software applications (esp. proprietary ones) have.
Yeah, honestly it kinda sucks that we're adding this layer of inefficiency and bloat to things, but these tools were invented for a reason.
This is because the Linux model of global system wide shared dependencies is stupid, bad, and wrong. Docker and friends are a roundabout a way of having a program shipping its dependencies.
It doesn't work very well for 3rd-party software distributed separately from the OS distro and installed by end-users.
The problem I've seen is that, while pre-Docker there was really nothing preventing ISVs from packaging their own versions of dependencies, they still only targeted specific Linux distros and versions, because they still had dependencies on things included in that distro, instead of just packaging their own. The big thing is probably glibc.
As I recall, Windows went through a lot of similar problems, and had to go to great lengths to deal with it.
Literally not in the Top 1000 problems for modern software.
> Windows went through a lot of similar problems, and had to go to great lengths to deal with it.
Not really. A 20 year old piece of windows software prettt much “just works”. Meanwhile it’s nigh impossible to compile a piece of Linux software that runs across every major distro in active use.
No, it only works because Windows basically included something much like WINE (they call it WoW) in Windows, so old pieces of software aren't running on the modern libraries.
>it’s nigh impossible to compile a piece of Linux software that runs across every major distro in active use.
Sure you can, with Docker. It's effectively doing the same thing Windows does with WoW. Windows just makes it a lot more invisible to the user.
I've heard it said the greatest trick ever pulled was the devil convincing mankind he didn't exist.
The second greatest trick, apparently, was Google convincing the world to adopt a solution to a problem few other than Google actually have.
We have an open source library called Transact[0] that you can run anywhere, including locally, and get durable serverless with state (and even some observability). Then you can deploy it to our cloud[1] and get reliability, scalability, more observability, and a time travel debugger.