Validating Kubernetes YAML for best practice and policies | Hacker News Reader

Validating Kubernetes YAML for best practice and policies

learnk8s.io

83 pointskiyanwang54 comments

> Config-lint is a promising framework that lets you write custom checks for Kubernetes YAML manifests using a YAML DSL. But what if you want to express more complex logic and checks? Isn't YAML too limiting for that? What if you could express those checks with a real programming language?

Having recently worked a little bit with YAML for Kubernetes and HCL for Terraform, I really wish they had both just used "a real programming language" right from the start. I'll choose Racket because I know it best, but there are probably many languages that would work well. You could expose very nearly the same configuration language, but backed by a real programming language. I bet this would make some of the tools the author lists at the end (eg copper, config-lint) much easier to write, or perhaps not necessary at all.

And the author didn't mention Helm, but I will. The part of Helm I saw seemed to be a lot of work just to add "functions with parameters" to Kubernetes YAML, something we could have had for free using "a real programming language" from the start: https://helm.sh/docs/chart_template_guide/functions_and_pipe...

Why are so few configuration languages not backed by a real language?

> Why are so few configuration languages not backed by a real language?

In many cases, not having a full featured language is helpful as you have some additional guarantees that comes with a non Turing complete language like guaranteed completion.

In some cases though, you do need a full pledged programming language. For those cases, HashiCorp recently announced CDK support for Terraform: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-py...

Cool, I hadn't seen CDK for Terraform before. But doesn't this support my point? If they had just used Typescript from the start, they wouldn't have to add Typescript support later.

I suppose guaranteed completion matters if you are running untrusted code, but wouldn't sandboxing solve that? Are there any other guarantees that sandboxing wouldn't solve?

> I suppose guaranteed completion matters if you are running untrusted code, but wouldn't sandboxing solve that? Are there any other guarantees that sandboxing wouldn't solve?

There is more benefits than just that, by restricting the possibilities you know there won't be unbounded loops, analysis and code review is easier (and infrastructure teams are often seriously lagging in this regard), it can be easier to maintain, update and test.

In some cases, you can have a project where this is seriously limiting though because you have some very complex and specific thing you need to express. For this you can use CDK. I would say both approach are complementary, not exclusive.

In my experience I would say nearly all infrastructure projects can be expressed as Terraform rather easily, but YMMV.

You can bring that argument, but only until you decide to use YAML, instead of something declarative and simple like JSON.

Once you switch to something as powerful as YAML, you might as well reach for a real programming language.

Also, YAML is kind of a hack.

Living in Norway I've had quite a few of these situations where no is parsed to False [1].

[1] https://hitchdev.com/strictyaml/why/implicit-typing-removed/

It has been fixed since yaml 1.2

That said, yeah :facepalm:

> powerful as yaml

Can you make an example of the power of yaml that makes it more like a programming language (in particular vs JSON)?

not OP but: tags (custom data types), more builtin data types (dates etc), more syntax variations (heredocuments, multiple ways to write a string,a boolean, etc), comments, references+aliases.

The language is vast, and most people use it without knowing it.

I love YAML, but I wish there was a "strict&sane yaml subset".

All those are good examples of yaml complexity but none of that is anywhere near what even a simple programming language can unleash (variables, loops, functions, recursion etc)

You can do recursion in YAML, and it ends badly: https://en.wikipedia.org/wiki/Billion_laughs_attack#Variatio...

Recursion without termination doesn't produce a useful computation

Can a config mechanism where shell commands are embedded as strings still be considered as offering these additional guarantees?

Perhaps certain things are easier: it's easier to parse, to isolate the code portions, and to read data portions. But then validating what's inside those code portions is a challenge.

It seems like there's an opportunity for a programming language purpose built for the task of configuration. Its chief feature would be the ability to provide a lot of information when read in "inert data" mode, yet also provide full programming language power when read in "run" mode.

ETA: Perhaps embed Python inside something similar to YAML, and support active code via a "lambda" type. Use indentation for delimiting.

   project: "Foo"
   version: "1.0.1"
   email: "me@example.com"
   load:
     lambda file_path:
       pass

Because when you do that, things always go astray as not everyone is that disciplined in what they put on those files.

Tcl, Perl, Python based config files experience in production, means I only touch Gradle when Google forces me to do so on Android.

Everything mentioned in the article can be simply done by writing a kubernetes validation webhook in the language of your choice. Why would you specifically need the configuration to be a real language?

This so much.

Every time you have to use another configuration language you have to learn all the quirks (ex: "" == undef in puppet), Give up all your powerful tools (like a debugger) and learn new abstractions. It's extremely counterproductive and it usually would have been better to have simple data structure that you generate from a program written in a traditional language (which the team can artificially restrict to avoid recursion if they want.)

Also IMO there's no such thing as a "declarative language." These are just languages where almost everything has a side affect of mutating a data structure that you don't have an easy way of inspecting or debugging.

> I really wish they had both just used "a real programming language" right from the start

There are currently several options, like https://github.com/stripe/skycfg

> Why are so few configuration languages not backed by a real language?

There are tons of tools that work with YAML/JSON but I imagine you mean more like a first-class citizen programming language specific or good at writing static configuration. The other side of the coin is that configuration in many cases have a different audience than programmers (for ex, end users or operations team) and it's a good option to have static configuration key=value than any human can (more or less) read easily without having to run programming code in your head. Plus programming, besides being harder to read and share, introduces bugs. So I suppose the preference between code writing configuration and stand-alone configuration depends on who the consumer is and how complex the configuration is.

That's one of the things I like about chef. You have an extensible DSL and templating but you can also write Ruby if needed.

k-rail does realtime validation and enforcement with policies written in Go: https://github.com/cruise-automation/k-rail

using a "real" Turing complete language for configuration is overkill.

But there are two promising projects that could finally make a difference (at least one of them).

https://cuelang.org

https://dhall-lang.org

I'm having some arguments with other developers (devs) on whether or not this is important. I'm gonna finally try to implement this for my own pipeline this week, hopefully.

I would much rather have devs double check/validate things locally before they edit changes.

Modifying config files by using the edit text feature in GitHub (GH), doesn't enable you to do that.

& Devs are lazy. I'm lazy. They want things easy. Me too.

So let's make it easy. Modify your CI/CD pipeline to validate YAML configs on any file changes (use GH hooks for example)

Now devs can do whatever they want - if their pre-deployment checks fail, go back and fix it!

This is a very sensible approach. One pro of having the checks automated instead of just having the developers check carefully their changes is that onboarding a new developer is easier, you will spend less time on very small and specific details and you won't forget to tell some detail.

This is a good approach because it focuses on the desired outcome ('no invalid configs get deployed'), and doesn't try to use a proxy ('you have to validate locally') to get there.

You're basically describing Sentinel for Terraform (https://www.hashicorp.com/sentinel/) or Datree for Kubernetes (https://www.datree.io). There are also a bunch of tools popping up in this space that focus on catching security issues rather than misconfigurations.

You are using YAML already. How much do you care about best practices.

Only partially kidding here.

You're currently being downvoted, but I agree, YAML is kinda terrible, not sure why anyone thought Python's syntactically relevant whitespace was ideal for a config file.

Classic example:

  - containerPort: 7173
    name: http

I think that's an object in a list? But it's not overly clear.And if I indented any of those lines wrong...

also, keywords of the underlying software mechanisms are decoupled, so uncheckable for plain YAML tooling. Whereas with XML you can at least infer a lot about the desired structure and keywords from the schema. Deeper checks are only possible with "real" programming languages, preferrably statically typed ones. I'm wondering when that wisdom trickles down to configuration languages.

Don't forget the plethora of truthy values that put Javascript to shame.

   - country: NO

Of course it's boolean

that was changed in YAML 1.2[0], which I think is many years old, but people don't specify a version on config files anyway and I'm not even sure most parsers respect it, so it keeps popping up, which is sad.

[0] https://yaml.org/spec/1.2/spec.html#id2803629

That's the problem with bad formats: they live forever. They and any number of parsers for any version between version 0 and now.

>not sure why anyone thought Python's syntactically relevant whitespace was ideal for a config file.

Because it reduces syntactic noise and increases informational density.

Most other serialization formats end up putting in indentation anyway to make them readable (JSON, TOML).

That's a map as a list element, iirc the terminology. But check out this.

When studying the Yaml spec I discovered that a map property (key: value) can have not only a string as its key, but any value. Even a list. (cue screams)

Here's another reason why YAML is bad:

the fact that I have to read the spec to figure out what's going on.

It should have been fairly simple to do this.

Why do I have to know the difference between ":" and "=" ...?

Why didn't anyone learn from CSS when we all made pre-processors like stylus, less and sass that added white-space sensitivity. It's not great.

I thought HCL was one of the best attempts at a terse config language. Shortening foo { bar { bazz {} }} to foo bar baz{} cleans up plenty.

YAML is unusable without a sufficiently capable linter. Syntax is very fragile and completely decoupled from function.

You're partially kidding but you're 100% right.

Someone in another thread mentioned using some LISP or LISP-like language to solve this problem.

Would something like TOML make things better here?

What would have been the solution?

XML, but I guess I won't get many friends for asserting that.

Nope. Json has schema validation too, and every language has a json parser

JSON doesn't do comments.

Not officially, but it's fakeable

Which for me is a design smell, why bother with workarounds when XML already offers me all the best tooling out of the box.

For me, xml is just too much, and when working with C or C++ (not t sure about rust) it's just a pain. Can your json schema file not serve as the documentation?

Why should it be a pain? Parsing XML is no different than parsing JSON, there are plenty of robust parsers available since around 2000.

Mature libraries with plenty of experience deploying into production.

Plus for small enough stuff, like configuration files, one can just plug XPATH as well, making the search for items even easier.

If something like k8s implemented comments as specific comment fields that would actually be pretty useful. The fields could be parsed and show in GUIs.

That is the workaround, just add it as "comment":"comment text". Having one comment for each field in an object would get unwieldy though. Anther place for them is possibly in the json schema file.

Then use HTML and extend the functionality with JSX

Useless work when XML already has all the tooling available and battle tested in production.

I really regret disliking XML all those years ago, now that I've seen the alternative.

Dhall is an example of a configuration language. Its programs must terminate. It aims for safety, claiming that it can support safe evaluation of untrusted code. https://dhall-lang.org/

TBH, I have no experience with it. But, it sounds like if you need a configuration language with programmatic features, it would be more suited to the job than a general purpose programming language.

I recently had a bug that wouldn't have happened if I'd had these in place: https://dev.to/darklang/a-fun-bug-55cl. I added similar checks (and kube-score and polaris seem like good tools - I might try adding them).

I think that this is a great approach to test out the files. Mistakes in those files can cause a production outage. I like doing those tests once a PR is open and before it is merged into master and executed on the production cluster. (Disclaimer i am a co-founder of datree.io)

And so the circle of schema validation does yet another turn, now back to those XML config files.

Every time I see a JSON config file I miss those kilobytes of XML.

I find Intellij's K8s plugin really helpful for identifying issues within a single K8s YAML file, but it won't find things like a deployment.yaml without a pdb.yaml but it's a good start.

Reply on news.ycombinator.com