Docker-compose.yml as a universal infrastructure interface
ergomake.dev
ergomake.dev
Even their example of using Ingresses is already leaky because there is no similar concept of Ingresses in docker-compose, so they're hacking it with labels. What if I need to set a higher request timeout for my app? I have to set a docker-compose label that converts into an Ingress annotation (the leaky part of the Ingress abstraction)? And this is somehow less cognitive load?
Like all similar attempts to simplify deploying apps, it only works if all your apps fit in a nice box. Anything slightly non-standard (like needing different classes of storage volumes) will expose the leakiness. Even Kubernetes kinda still expects all your apps to be stateless web services and will fight you otherwise.
Could you give an example? I thought k8s and its YAML/practices was the standard?
in my own experience, i would've thought "ok, let me aptinstall k8s and then apply some yaml to it so i can avoid going to the cloud for a massive k8s bill for almost no reason"
standing up kubeadm on a single master + worker node setup on a VM (like a droplet or EC2) is basically like, extremely non-standard + frowned upon
then, does anybody actually write pure k8s yaml? or do they write jsonnet/kustomize/helm charts? do they apply with kubectl or the helm CLI or the argocd CLI?
You can think of plain k8s yaml as a sort of machine code for k8s. You _can_ write yaml by hand, but it’s not generally how things are done outside of initially learning how k8s works.
The tools you’ve mentioned all make working with k8s resources a lot more pleasant. Something simple like parameterizing the name of a secret instead of hardcoding it in N different yaml files saves a lot of time if you ever need to refactor, and being able to provide those values via a “standard” CLI tool makes automated deployments a lot easier compared to hacking together some yq commands. helm in particular tracks the history of any charts you’ve installed, and has a very simple “helm rollback” command which can get you back to the last working version of your application if things end up going wrong.
The exact tools used by each team will differ; you could use any of them. But no one in my experience is writing plain yaml manifests and kubectl applying them in production contexts.
For example try to configure a udp connection in k8s on aws.
> Once you get to the non-standard, cloud specific configuration
what's a common use case for when this is hit?
Contrast this with the contract* K8 tries to sell you, which is "describe the intent and we'll figure it out for you" and you'll realize the cloud providers bunged up this whole thing.
That's not to say K8 is perfect, but if you think for even a second that wiring up resources outside your kubernetes cluster will be lower-friction, kubernetes is falling short of it's goals.
One part I haven't quite put a handle on is where/when to use CRDs in either helm or OLM. The community still seems split on how to use CRDs.
That may just be unique to my job. Maybe another company could force all apps into nice boxes that prevents their bespoke platform API from leaking.
New team member: "How do I get the stack running?"
Existing member: "Clone this repo and run docker-compose up -d"
20 minutes and a cup of coffee later:
New member: "Cool! Ready to start!"
Does it look like our AWS stack? Only superficially. Which can be a good thing. The compose file shows you the broad strokes, not the extreme details. Compose files help with the basic mental model. Want to know more? Read through the CDK stack code.
But using a compose file for a cloud deployment interface? No thank you.
And then starts the hunt for tradeoffs, with a million pitfalls.
I have seen the pattern fail too often to be only optimistic on the approach. I now tend to favor ephemeral deployments with only one locally running service connected to a bunch of remote services.
The compose.yml runs everything on a docker network and uses redpanda for the kafka broker. Iteration became SUPER easy. Previously, we were just writing config files locally and uploading them to S3, then restarting the dev instance of whatever we were working on in k8s to re-ingest. This was, as I'm sure you can tell by now, tedious
Has anybody found a better way with just open source tools?
Is anyone using Cue or dhall in a real production scenario? I'd be curious to hear reviews of those, as they look promising.
BLUF: Infrastructure seems to live in this weird "configuration code" space between pure config (YAML) and actual code. You want reusable config, but you don't want to have to understand the complexities of a real codebase. Jsonnet, Dhall, and CUE all have the simplicity you want, but we found the DSL we'd need to write to be rough. Starlark hits this very nice balance of safety and ease-of-readability (and as a bonus is basically just Python). We've been very happy with it so far.
Not satisfied I used a simple source filter to get a YAML compatible variant (WIP): https://github.com/elcritch/ants/blob/main/tests/cimport_exa...
Our concept of configs languages needs to keep evolving.
There are a lot of downsides though, because now you've got a leaky config language instead of a leaky config interpreter.
I toyed around with dhall to generate k8s deployments. It was fantastic, but then I realized there was no chance that others would be willing to learn FP. Cue is pretty great, but it still has many sharp edges in the tooling.
Ultimately, though, all configuration files tend toward turing completeness. Just use an actual programming language. If you need to dynamically parameterize it then you can import a JSON file or whatever.
As annoying as the parsing quirks of YAML are, they don't bite me that often. What bites me is the proliferation of YAML files in an effort to DRY my config files with whatever primitive imitation of a templating language I can use as the common denominator of the various tools consuming them.
I just checked our monorepo, and we have ~93 YAML files related in some way to DevOps.
Having to actually do the dangerous things that you typically do with YAML in order to test the YAML is insane!
This is the first time I've ever seen someone say this. Why do you think it's better?
The problem is that the two often conflict with each other, because I'm writing code to parse config written by a human.
My rule of thumb is that any autogenerated communication between services can be done over JSON (assuming some more efficient serialization format isn't available), but if a human needs to maintain it (and check it into a repo with diffs) then it should be YAML.
TL;DR: It has way too many opinionated things builtin, too many ways to do the same thing.
As a additional note, unlike JSON that has the awesome tooling available (`jq` is reasonably widespread, you can count on python/perl/node being almost everywhere), it has nothing that (because of the multiple opinionated ways) will reliably read your files or not mangle your desired output in some way or another -- except if you limit your YAML to just a embedded JSON.
I want to find out with CUE's BCL heritage to fix what they got wrong with BCL at Google being used by Borg, what the CUE team is doing to try to avoid the Second System Effect biting them as they build CUE.
Both CUE and Dhall [3] seem promising when first deployed, but some field experience is reporting challenges I typically associate with immature tooling ecosystems surrounding them [4] that come up short when trying to reason beyond what the out of the box experience can quickly support when working at scale.
One part I haven't quite figured out with these configuration expressions is how to seamlessly flow their models between organizational data models (to answer who owns what type questions, example) and operational data models (to answer who is on call type questions, for example), and gracefully manage the mistakes inevitably made in those models, without mountains of manual toil and bespoke in-house glue code.
[1] https://cuelang.org/docs/about/
[2] https://www.pulumi.com/blog/extending-pulumi-languages-with-...
We had DevOps for a while, but now systems have gotten so complex (and our tooling hasn't kept up) that we're back to pre-DevOps days with Dev throwing stuff over the wall to Ops.
Kubernetes may be a system that invites overcomplication, but it in no way requires it— in a world where anyone can run k3s or microk8s on their own laptop and a Kubernetes hello-world is like 10 lines of yaml, I don't think it's a stretch to posit that that's the interface that makes sense to use up and down the stack.
Agreed... Though I think there is a different solution. Instead of coming up with all these hardcoded formats and standards (docker-compose, k8s, etc, helm, ansible) which essentially are just 'functions' that map X to Y.
Instead of defining 'docker-compose' files, just define the containers itself. And if required, map that to a docker-compose, or k8s manifest or whatever. Remove all these unnecessary layers that aren't really that helpful - because each standard or implementation has it's own parser, error handling etc - why bother with all this?
And just use something like Nix for this. Or just define it as a common python value. Otherwise we end up building all sorts of templating and ugly programming-in-yaml. You could take a look at https://docs.hercules-ci.com/arion/#_arion_compose_nix to see what this might look like - uses Nix to do something like docker-compose.
https://docs.docker.com/engine/context/working-with-contexts/Of course each implementation is opinionated, AWS for example is deploying to ECS with ALBs.
And it is all good for a list of services, but when you add configuration it becomes a mess:
secrets:
foo:
name: "arn:aws:secretsmanager:eu-west-3:1234:secret:foo-ABC123"
x-aws-keys:
- "bar"
So to keep the interface clean you would need to split it up in a Service model and a Deployment model that exposes the needed configuration for the target platform plus the service configuration for the specific environment.
Or something...
Thinking of something generic that works in all cases is hard and complex...and opinionated hence the many platforms, I think.
I posted this a while ago https://news.ycombinator.com/item?id=34225669
The hardware has finite resources, and the best way to share them is to run things natively on the operating system.
It's also all simple and straightforward, everyone understands ssh and simple Linux sysadmin.
Unfortunately most applications are developed in a way that they believe they are the only one on the system. Ruby gets called, and they expect 2.7.
And there is another one which is using Ruby 3.2. And it's incompatible with 2.7.
Now what? Do you want to spend time
- maintaining 2 versions of ruby on a system
- keeping them separate
- ensuring they don't mix up
OR
Shove them in a container, run them in parallel and all they surface is a port.
I'll take the container.
Docker containers, in my experience, have not figured out auto updates yet. You may be running containers with critical system vulnerabilities and is not bothered to check.
So the basic premise of Dockerization 1.0, "I have exactly one thing which I will put in a single container with all of its dependencies, reproducibly built" is wrong.
Example: 2 containers which rely ffmpeg, and there's a vulnerability in ffmpeg.
But container A can't update cause ffmpeg's update breaks something else...
At least with Docker you can update container B, and maybe isolate A.
On a direct install ffmpeg is a shared component. So now both apps are locked to a vulnerable version.
Not to mention that shared components are often outdated for the sake of stability.
So App A is restricted to the same version as App B, even though App A would like version 9, and B is on 8.
In the Docker scenario, it's quite often that neither A nor B will be updated for a long time, since there's no mechanism to do so.
In general it's better to have fewer and more powerful and reliable nodes that you own than to have many disposable small ones, as shared instances just don't work well and aren't cost-effective.
Cloud in general just makes everything more expensive while removing control. The costs on AWS if paying for three years upfront is the same as buying the hardware outright, except you have little say in what the hardware is and how the infrastructure is set up.
> as shared instances just don't work well and aren't cost-effective
of course ymmv from mine, but this statement being true or false depends on too many things to be of any value
> as shared instances just don't work well and aren't cost-effective.
goodness you are outing yourself here aren't you
> It's also all simple and straightforward
What you're really trying to say is that cloud technologies are complicated for you.
If you are not running in prod in Kubernetes, but you still run in prod with Docker, what the heck are you doing? ECS? You still need a control plane to deploy the images and Compose isn't really up to that task for production workloads.
Terraform is just as easy, but probably a combination of the two docker-compose and terraform are much more effective.. Universal Infrastructure interface is the name of Terraform's game.. They already done what you hoped docker-compose did.
Terraform deserves serious kudos for interoperation between all the clouds, docker, vmware, and literally tons of API modules.. Their Infrastructure Modules are nothing to laugh about in a JSON compatible example.. Serious, they even give Ansible a run for their money when Red Hat was all the blaze.
I do wish them luck though.
I would not use docker-compose.yaml for that, though.
https://www.techtarget.com/searchitoperations/news/252474360...
On the other hand, you have something like CloudFoundery on top of K8S:
So if your developers can't learn this stuff...
A Deployment, a CronJob, a Secret, etc.: those are the interfaces with Kubernetes. By extension, it is the interface with which you interact with the service (a cluster of machines) that your infra team maintains. (And they are the ones that need to understand how to maintain & operate Kubernetes.)
App devs need to learn the structure of those YAML interfaces. They are incredibly well documented, both in reference material and in example material.
And yet I find the same as the people above you in this thread: app eng have an absolute aversion to wanting to specify, in programmatic form, how to run their app. Yet, by definition, they're the only ones that know how to do that.
(That said, I have been known to reverse engineer applications enough to understand how they intend to be run, and then write the corresponding k8s YAML. This is an organizational anti-pattern, though.)
The YAML isn't complicated. What's complicated is that the set of data that comprises a process is inherently complicated: what user does it run as? what files does it need? what command to execute it? what env vars? etc. k8s's YAML does rather little abstracting over these.
(I sort of disagree with the "fire them" — rather, you need to ensure your onboarding trains them. But a lot of employers these days seem to do zip for training.)
They might very soon. https://archive.ph/YWp4O
K8s includes a lot of Unix. If you don't know what a user or perm is, or a mount point, you just blame k8s.
You have skipped over the Pod and ReplicaSet abstractions, and the Pod part is definitely not ignorable even in the most trivial case. The container part is probably more ignorable than the pod part, since pods are what actually gets exposed for most operations.
No - fire their managers, and their managers as needed.
By definition, that's where the responsibility lies. Not with the leaf-node developers. Who are doing exactly as they've been trained and/or vetted by their managers.
I hope the people who manage your k8s infrastructure know how to develop the application they are deploying. If not, just fire your k8s teams. You cannot became k8s person without knowing how to develop the application, there are more interesting areas...
That sounds dumb, isn't it?