Kubernetes YAML Generator
k8syaml.com
k8syaml.com
Do we need a better layer of abstraction, i.e. better adoption and tighter integration for something like kustomize? Have we fucked up completely with Kubernetes due to it being outrageously complicated for simple tasks? How we redesign this to be simpler? Is the complexity even a problem for the target audience?
I've no idea. I just know I'm a kubernetes admin and I can't write a deployment yaml without googling or copy/pasting.
Our goal is to make a Heroku-ish platform for getting an app online - but one that doesn’t hold you over a barrel later on. You can even host from a spare home server :)
(Disclosure: I’m the CTO)
I'm interested in/have previously given up on a product where the control plane is managed for me, I can join the cluster with bare metal nodes, and then it's just Kubernetes.
Right now we don't do "hosted control plane", rather - you can either use our hosted clusters where you have namespace-level access (so you still get to use Kubernetes, just not all of the resources, for example, no `Nodes`), or you can attach your own cluster to our UI (with the advantage that we can setup ingress and forward traffic to you, which is perfect for a home-hosting setup).
Hosted control plane is something im watching closely though - I'd really love to offer that as a service, but since we're small, we're trying to focus hard on a core offering. Will be considered in the future though!
That is why in many businesses there is an OPS team managing the Kubernetes and providing tools like Cert-Manager, Istio, ... and the rest of the company who just use what the OPS team made.
Right now, everyone is building its own distro, proving IMHO the need for it.
I do however have plans of supporting enterprise by packaging a version of the app into an on-prem deployment system so you can just deploy your own kubernetes cluster and install the app and have your own enterprise version of Primcloud.
I will over time expand on the content, but I'm focusing on the platform itself over marketing stuff.
All this is to say that you're at very least certainly half-correct, in that k8s is a very flexible tool that can be used to build a very simple, elegant, and ergonomic PaaS.
I'm not sure I agree that it's inappropriate for most businesses though, unless you think that only a PaaS like Heroku is appropriate for most businesses; the analogy I'd suggest is "Heroku vs. running your own VMs" circa 2010. Heroku is great for getting started, and lets you move fast by abstracting away a bunch of infra. But it's also restrictive; you can't pick and choose your components freely. As you grow past a certain point you'll almost certainly need the flexibility (or just cost-effectiveness) that you get from running your own infrastructure.
K8s is an improvement here because you can run a managed cluster on something like GKE, which takes away most of the operational toil, while still giving you a lot of flexibility on what components / pieces to include. The k8s domain API does a great job of abstracting away true infra concerns like volumes, compute, scheduling, load-balancing, etc, while making it really easy to package, use, and iterate on the stuff that sits on top of that infrastructure.
I'd probably not encourage a seed-stage startup to use k8s unless you're very familiar with the tool; a PaaS like Heroku would likely be more appropriate. However at the point that you'd usually graduate to running your own VMs (wherever that is on your own company's developmental path), I'd say that using k8s is now a better choice.
> However at the point that you'd usually graduate to running your own VMs (wherever that is on your own company's developmental path), I'd say that using k8s is now a better choice.
However, even then there are intermediate options between VMs and full-on k8s, such as AWS ECS/Fargate, a Kubernetes distro, or managed Kubernetes offering (e.g., GKE) which give me the flexibility to interact with k8s, but they come with sane, pre-configured (or easily configured) solutions for logging, monitoring, ingress, load balancing, upgrades, etc.
From light googling I only see examples of building gRPC APIs and microservices.
It's good to know that such an interface could conceivably be built for Kubernetes, using the HTTP API. Even better that it supports any programming language (since it's a network interface); I'd see that as a step-up from aurora configs, in fact.
At some point, I imagine this will become the default way that Smart Organizations build around k8s, once k8s's core API is super-stable and people develop sound frameworks in $POPULAR_LANGUAGE for declaring your configuration.
Then, we can then banish yaml to the same fate as json and xml. One can dream, at least. :)
Funny you mention this, check out what is effectively this but for Kubernetes:
SMART_ORGANIZATIONS=$SMART_ORGANIZATIONS:stripe
I wonder if they were inspired by aurora, it's very similar.
I wouldn't be surprised if we eventually see new abstractions for "you just want a plain 'ol deployment with CI/CD du jour, a persistent volume claim, and a service ingress, just like 90% of all other CRUD webapps? Sure, here's a simple resource for that."
I think we'll start seeing a move towards more "opinionated" tools, just to outsource some of the decision making. No sense learning how to write your own pipelines if you can find a tool that says "we're gonna deploy every time you make a git tag and run `mvn package`, you figure the rest out".
The rest are soon to follow I’m sure.
Don’t get me wrong, I actually love tekton because I hate everything that is Jenkins and most other CICD tool integration with k8s. So tekton is just amazing in that regard.
But there is a huge learning curve for people who are used to old school tools like Jenkins and its pipelines .
* `helm create` to get the default scaffold
* modify a handful of entries in the generated values file
* done!
Only thing is the default helm chart starter does not allow for autoconfiguring of volumes, and since we're porting a lot of stateful apps to kubernetes we just modified the default starter to include that capability.
Of course it would be nice to not have to maintain a bunch of different virtually identical templates.
Pulumi looks interesting but unfortunately seems to insist on vendor lock-in (see the jerk-around on https://github.com/pulumi/pulumi/pull/2697). So I'm looking forward to the AWS CDK (https://aws.amazon.com/cdk/) maturing a bit.
Some potentially useful tips:
* One docker container usually maps to one pod, which is created by a deployment. You can put multiple containers in a single pod, but this couples them together tightly.
* Use services to assign hostnames to pods
* Have pods communicate to each other using the service names. This works the same as putting them in the same docker network.
* Volumes depend on your cloud provider or if you're running on bare metal. In the cloud it's easy, you just request one and it gets created on demand and backed by a cloud disk.
* If you're using volumes, you probably want to use the Recreate updatePolicy for your deployment. This will ensure the old pod is shutdown before creating a new one. Which is necessary to work with block volumes.
When using helm start with a `helm create CHARTNAME` and take a look at what it generated for you. You'll get some heavily templated yaml that if you're lucky you will barely have to touch.
But it's best to go through these tutorials and learn how to use the basic building blocks of kubernetes directly: https://kubernetes.io/docs/tutorials/kubernetes-basics/
Once you're familiar with what a Deployment, Service, Ingress and PersistentVolumeClaim are you can use helm to template this for you where necessary.
Collectively, as an industry we have gone insane.
I really hope we move more towards these opinionated tools that can handle 90% of the use cases. Most people just want to host an app on a port, and it's a pain to have to develop that pipeline at every company that wants to adopt Kubernetes.
Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier?
I seem to remember a lot of wrangling custom kernels to get Asterisk sounding just right, bizarre Apache, & network configs.
It’s just text? It’s always going to turn into a nebulous mess without literal edges and boundaries.
That’s Google’s play with it, IMO. Train tracks. Which is what I hate about it.
Google hasn’t built a less Byzantine text mess. It’s built hype though, with a boring tool
I don't really get the love for hating on K8 complexity on here - nobody says it's the perfect tool for every use case... but when you do need it, it's amazing, and has many advantages over the current trend for serverless IMO.
> Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier?
Yes, and by far. Adding a layer on top of all the traditional Linux daemons, tools and libraries does not decrease the total complexity - quite the contrary.
When you have a bug in an application that is related to something in on another layer you have to walk through the whole stack.
Examples: A bug in a network card impacting only large UDP packets. A race condition of file access triggered by NFS or a storage device driver. A vulnerability based on a timing attack due to CPU caches.
The deeper the stack, the worse.
I find the bog-standard Prometheus chart provides me a pretty incredible level of monitoring out of the box, usually it’s pretty easy to pick the bad one out of a graph.
Running your own VMs without something like k8s? Yeah this setup I can deploy and have working in an hour is gonna take you a week to set up properly. Standardization is valuable. Abstraction is valuable.
No. Read my post again: I did not wrote about hardware issues.
Most work around optimization, reliability or security require digging through the whole stack sometimes down to the kernel.
> When I find a node with issues, I can just delete it from the pool and get a fresh one back.
However, a lot of k8s deployments are on-premise, where you have to debug your own hardware.
> The bad network card? That’s for Google/Amazon/DigitalOcean to deal with.
First you have to pinpoint the root cause of that glitch affecting all the containers running on VMs using the same bad drivers. Often it could be the same in 50% of your fleet.
> is gonna take you a week to set up properly.
Most certainly not. I've been deploying large production fleets in minutes since 2005.
It provides the promise of portability, so the business feels less locked in to a vendor, while still being dense and impenetrable enough to them that IT won’t be done out of a job.
Simpler, more efficient alternatives are seen as more expensive, because IT hides the true cost of just how much of their salaries is spent of non-productive pottering around with yaml.
These systems and products also threaten to automate IT out of a job and don’t necessarily count much for their resumes. If you’re not a programmer what interest do you have in dumping all of your intricate, fragile environment for a PaaS? Not much
There's trade-off between a tool being too specific to a usecase vs being too generic with the associated "boilerplate". But I don't know how much simpler Kubernetes could be made while still covering the intended scope?
It has very simple basic model, which allows you to recursively built more and more complex abstractions on top of it.
More tooling to write good k8s yaml sounds good to me.
[0] https://datree.io (Disclaimer: I work with them.)
What about Nginx or Apache configuration files? Could you write them without googling?
Easily, because noone writes them from scratch, they just modify the default one.
For most of my Kubernetes work I "kubectl describe" something existing into YAML, modify it, then "kubectl apply" it back again.
kubectl create deployment my-deployment --image nginx --dry-run -oyaml- a dedicated editor with intelligent autocomplete
- stop using YAML, it soon becomes unreadable. JSON is easier to grok.
Best way is to stop using both, and generate the objects from higher level language, at least something like Jsonnet (which is really just a step up, so better not stop there but I will take what I can)
I'm on the fence about this, something like Dahl seems good, but I am not a fan of using fully-fledged higher level languages (the Pulumi approach) as I've been burnt before with devs writing bad code and generating configs in the most confusing, convoluted way possible.
For anything non-trivial you will want inline comments.
Also while any JSON can be expressed in YAML, the reverse is not true.
Yes, JSON needs comment support back.
> Also while any JSON can be expressed in YAML, the reverse is not true.
This is a feature not a bug.
A.K.A. a schema file that a general purpose editor can consume. Please don't make me use some single-purpose editor just for autocomplete.
What Kubernetes allows you to do is very complex so you need a capable system to express it all. YAML isn't always the best but it works fine for most and tools like these are very helpful.
Do you use an IDE? Does that mean the language and framework is too complex? No, it's just a tool to help you get things done. More tools aren't a bad thing.
This is simply not true. Most complexity in today's systems is completely avoidable. Most developers are just mentally stuck and don't even try. "Managing" complexity is a great way to achieve job security without becoming good at anything specific. Instead of learning how to design system people are learning how to write config files.
If you don't need Kubernetes then don't use it. But if you do then you can't make it any simpler than the complexity needed to deliver the functionality.
Built in deployments, process health, logging, load balancing, security, high availability, config and secret management, volumes and persistence, and much more in exchange for some YAML files is a pretty good encapsulation of complexity though.
Without going into the complexity of Deployments, consider the lowly Pod. What configuration does your app need? What is the name of the container that contains it? How much memory does it use? How much CPU does it need? What ports does it listen on? What HTTP endpoint handles the health check? Does that endpoint test liveness or readiness? What filesystems does it need? What setup needs to be done before the main container runs? Does it need any special resources like GPUs? The list goes on.
The problem here is that when you're writing a Pod spec, you're building a single-purpose computer from scratch. In the traditional UNIX world, people answered most of these questions for you. How much RAM can my app use? However much I plugged in. How much CPU can my app use? All of them. What ports does it listen on? Any of them from 1024-65535. What filesystems does it need? Whichever ones I setup in /etc/fstab.
I don't think it's a stretch to call UNIX's "yolo" approach problematic. It is great when you have one server running one app, but servers have gotten gigantic (with pricing to match) while applications have largely stayed the same size. This means you have to pack multiple apps onto one physical server, and to do that, there have to be rules. When you write a Kubernetes manifest, you are just answering every possible question upfront so that the entire system runs smoothly even if your individual component doesn't. It's the cost of having small apps on big computers.
The problem comes from applications that you didn't write, or don't fully understand. Before you can understand how the application behaves, you have to write a manifest. But you don't know the answers to the questions like how much CPU you're going to use, or what the worst case memory usage is, etc. This causes a lot of cognitive dissonance, because the entire file is you admitting to the computer that you have no idea how to configure it. No abstraction layer is going to fix that problem, except by hiding those uncomfortable details from you. (And you will always regret using the "yolo" defaults -- who hasn't tried to SSH into a broken server only to have Linux helpfully OOMKill sshd or your bash instance when you're just trying to kill your malfunctioning app.)
This is largely the fault of application developers. They aren't willing to commit to reasonable resource limits because they don't want to handle support requests that are related to underprovisioning. My experience is that applications that set limits pick them wrong. For example, GCP and DigitalOcean's managed Kubernetes offerings both install monitoring agents to support their dashboards; these apps ship with limits that are too low and any reasonable Prometheus installation will notice that they are being CPU throttled and warn you about it. Now you have to waste your day asking "is this a real problem?"
Many open-source apps go the other way and pick resource limits that truly encapsulate the worst case and require individual nodes that are many times larger than the entire cluster. Yes, it would be nice if I gave each pod 32 CPUs and 128GiB of RAM... but I don't want to pay $2000/month/replica thankyouverymuch. (I've been on the other side of that where resources didn't cost me real money and happily used terabytes of RAM as cache.)
Application-level configuration is also not in a great state. Everyone tries to sell you their curated defaults so they don't have to write any documentation beyond a "quick start". (I'm as guilty of that as anyone in fact!) The application will have some built-in defaults (so the developers writing the app can just "go run main.go" and get the config they need). Then someone comes along to make a Helm chart for you, and they change the defaults so that their local installation doesn't need any customization. This only causes problems because instead of an undocumented underlying application, now you have that AND an undocumented abstraction layer. You may find the answer to your question "how do I configure FooApp to bar?" but have no way of communicating that config through the Helm abstraction layer because the author of the Helm chart never thought anyone would do that.
This rant has gotten quite long so I'll wrap it up. No abstraction layer is ever going to make it so you don't need to answer difficult questions. The actual list of questions to answer is available through "kubectl explain pod.spec" and friends, however.
Tools like Helm solve this problem.
Helm, like any abstraction, has costs and values. For example, it's very valuable to be able to encode something like "my app consists of a frontend and a backend that run in the same Pod and must each use the same version of the code":
apiVersion: v1
type: Pod
metadata:
name: foo
spec:
containers:
- name: frontend
image: my.registry.io/foo-frontend:{{ .tag }}
- name: backend
image: my.registry.io/foo-backend:{{ .tag }}
The problem with this abstraction is that there's no escape hatch. If you wanted to run different versions of foo-frontend and foo-backend, there is no way to say "but no, really, this time I'm violating the rules for a good reason". You've reduced the features available... the only way forward is to start over with nothing.
The result of this is that every individual Helm chart has to account for every possibility that the manifests could possibly encode, and invent their own programming language that is identical in expressiveness to the underlying manifests. And they do! Differently every time! For example, if I wanted to allow people to override the container images, I'd have to make my template look like:
containers:
- name: frontend
{{ if .tag }}
image: my.registry.io/foo-frontned:{{ .tag }}
{{ else }}
image: {{ .frontend_image }}
{{ end }}
- ...
I think we can all agree that didn't save anyone any time or effort.
This example conveniently flows into my other complaint with Helm. The "yaml files" that declare templates aren't actually valid YAML. You can't use something like `prettier` to autoformat them. You can't use the YAML language server to provide code advice as you type. You can't use `kubeval` to validate them. You are throwing all of that away to Build Your Own Thing. It is actually very insidious and for that reason I consider Helm to be more harmful than helpful. It isn't an abstraction, it's just a macro that might be good for one person the instant they happened to type something in.
The other problem is that Helm charts have no upgrade path. They are only designed for "please explode this project into my cluster, I promise to clean it up In The Future". It never gets cleaned up and brings a little piece of un-updated Windows 95 nostalgia right into your cluster.
Helm is actively harmful. And people love it, because it saves them a tiny bit of time one day at the cost of a lot of time in the future.
I see putting your example of two, unrelated, containers in the same pod as the same binary problem I mentioned earlier. I get your point, but it's a scenario one must wedge themselves into by making other poor choices. Why must the frontend and backend be the exact same version? The most obvious possible reason could be that the API used between them isn't versioned. Or maybe there's not even an API!
> You will have to read the code or hope I wrote documentation for my ad-hoc Kubernetes extension.
No, helm's approach to inserting variables into templates means this isn't the case. Every option and default appears in values.yaml and it's a one stop shop to see everything you can customize. The code example you wrote could be better written as:
containers:
- name: frontend
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
image:
repository: my_default_image
tag: my_default_tag
I'm not sure what you mean about helm charts having no upgrade path. IME you can un-deploy, upgrade, and rollback helm deployments and it takes care of adding/removing kube resources that where also added or removed in the yaml for you. [1]
Honestly, I get the pieces and I know where to look to get what so I'm not too bothered. The problem with wrapper tools is that sometimes I can't get at the insides and then I have to learn the wrapper tool. So I'm going to just stick to raw Kube until one of the wrappers wins out.
If k8s changes notation having a layer of abstraction (hopefully) allows me to run the same command to generate new YAML (very helpful in automation)
For production-y things however, some meta-config language that allows deterministic templateing would be a huge improvement. It allows you to make sweeping/uniform infrastructure changes from a single library or tool change.
Kubecfg is a good example of the basics one could implement [0] although it's examples aren't as fully fledged and organized as they could be.
[0] - https://github.com/bitnami/kubecfg/blob/master/examples/gues...
Although it is still in it's early days, it still is excellent to use and will only get better with additional tooling.
We can use TypeScript interfaces (which give us nice ide code completion) to define our yaml.
we can then create functions where we would normally duplicate Yaml. Really nice. https://www.pulumi.com/kubernetes/
This is the way to go for sure. I've done similar by generating CloudFormation from Python (I wrote my own library because I felt Troposphere was not very friendly nor a significant improvement over YAML).
Typing turns out to be pretty useful when you're generating YAML. While my library was fully typed, Python's type checking left a lot to be desired--many stupidly common things still can't be expressed (JSON, kwarg callbacks, etc), getting mypy to find/accept type annotations for libraries is stupidly hard, and the IDE integrations are pretty awful at this point. TypeScript users would enjoy a real leg-up here since its type system isn't half baked.
Yes and no.
Typing is a must, but a full-blown programming language is too powerful and all abstraction layers start to leak sooner rather than later. I always ended up with a "deployment" function that exposed almost all underlying functionality.
We're big fans of the Cue (https://cuelang.org) approach instead: https://cuelang.org/docs/about
I've looked at Cue a few times (principally out of frustration with the lack of types in Starlark), but I don't really "get it". What I want is a typed embeddable scripting language--specifically the ability to expressively generate YAML, and I'm pretty sure that's not what Cue is. I'm open to the argument that Cue is better than what I want (a la "if Henry Ford asked people what they really want, they'd've said faster horses"), but the value proposition isn't clear to me from perusing the docs. Maybe you can correct me?
Paul, as an aside, we absolutely love how feature packed Octopus is nowadays. We have been using it since 2.0, and I don't think we will ever give it up. Thanks for making one of the best tools we use a daily basis!
A UI like this is useful to so many. It makes the experience of creating these YAML files easier. Thanks for sharing it.
That said, this still looks cool. I just hope we won’t need a Kubernetes configuration generator generator anytime soon.
The bigger issue is that there's a lot of things you can do with it, and editing text YAML/JSON serialization of the objects is probably one of the least efficient ways of dealing with it, but it's also the "default" way people know.
It's much easier when your editor actually understands the object you're editing instead of at best helping you write correct YAML.
https://github.com/zegl/kube-score
https://stelligent.github.io/config-lint/#/
I'm obviously biased, but it's been hugely successful! kube-score is working very well out of the box, and there's only a handful of cases where the "ignore" annotation has been used to disable a check that's too strict for the particular use case.
Feel free to reach out if you have any questions or comments.
Aside from making it easy to generate k8s manifests, this could also be a great learning tool. If you allowed this to generate multiple resources that are linked, it could be a great illustration of how different resources fit together.
Clear schema (like TypeScript interfaces or something similar) which allows generating an UI.
[1] - https://kubernetes.io/docs/concepts/overview/kubernetes-api/...
[2] - https://github.com/kubernetes/kubernetes/blob/release-1.19/a...
[3] - https://godoc.org/k8s.io/client-go/kubernetes/typed/core/v1
[4] - https://kubernetes.io/docs/reference/generated/kubernetes-ap...
As a side note, usually the schema languages fail at some point, thus I referred to TypeScript interfaces, which are very flexible way to write validation.
There is already a schema support for JSON validation in VSCode, one can use it using `{ "$schema" : "https://example.com/url/toschema" }` but it uses JSON Schema format, which I think is not accurate enough in edge cases for UI generation.
The real power of Kubernetes is that it has very simple basic model that allows building complex constructs out of simple pieces, and in fact basic help (which is even used for UI in kubectl explain.
All the bits for good UI support are there.