cppforlife

i never found myself struggling with submodules, but at times i found myself just slightly annoyed (especially when having to remove/replace submodules), especially when they are used for simpler use cases.

i actually ended up creating https://carvel.dev/vendir/ for some of the overlapping use cases. aside from not being git specific (for source content or destination), its entirely transparent to consumers of the repo as they do not need to know how some subset of content is being managed. (i am of course a fan of committing vendored content into repos and ignore small price of increasing repo size).

since you mentioned Kubernetes...

> It would be nice if there was a separate state reconciliation system that one could adapt to use with Cue or Dhall or any other frontend

this exactly was thinking behind https://carvel.dev/kapp for Kubernetes (i'm one of the maintainers). it makes a point to not know how you decided to generate your Kubernetes config -- just takes it as input.

> In particular the ability to import other files as semantic hashes seems like a great feature.

it's an interesting feature but seems like it should be unnecessary given that config can be easily checked into git (your own and its dependencies).

i think kubernetes is not a great example in favor of more client state (like tf) since k8s has uniform resource structure (metadata.*) and first class labeling support. but as you point out kubectl doesnt use labels well (at least imho).

when building https://carvel.dev/kapp (which i think of as "optimized terraform" for k8s) the goal was absolutely to take advantage of those k8s features. we ended up providing two capabilities: direct label (more advanced) and "app name" (more user friendly). from impl standpoint, difference is how much state is maintained.

"kapp deploy -a label:x=y -f ..." allows user to specify label that is applied to all deployed resources and is also used for querying k8s to determine whats out there under given label. invocation is completely stateless since burden of keeping/providing state (in this case the label x=y) is shifted to the user. downside of course is that all apis within k8s need to be iterated over. (side note, fun features like "kapp delete -a label:!x" are free thanks to k8s querying).

"kapp deploy -a my-app -f ..." gives user ability to associate name with uniquely auto-generated label. this case is more stateful than previous but again only label needs to be saved (we use ConfigMap to store that label). if this state is lost, one has to only recover generated label.

imho k8s api structure enables focused tools like kapp to be much much simpler than more generic tool like terraform. as much as i'd like for terraform to keep less state, i totally appreciate its needs to support lowest common denominator feature set.

common discussion topics:

* whats the lowest common denominator for apis that need to be supported

* how much state to store client side vs server side (in the api itself e.g. tags or in "assistive service" e.g. s3 api)

* is it enough to just store resource identifiers vs whole resource content (e.g. can resource content be retrieved at a later point; if content is stored, is it sensitive)

* how easy is it to recover from complete state loss

hi, i've authored a set of OSS CLIs in this space based on my previous experiences orchestrating VMs, containers, IaaS resources:

- https://get-ytt.io (for templating/overlaying)

- https://get-kbld.io (for image ref updates)

- https://get-kapp.io (for deploying)

i would be interested to hear your take on comparison with your current pulumi setup.

i've been somewhat spoiled by using tools above so here are some benefits of using them that might align with some of your points within your article:

ytt:

- start with existing YAML configuration and "upgrade inline" with necessary templating

- fully sandboxed configuration building with a python-like language (no accidental dependencies on time, network, disk, etc. like you would experience with "regular" programming languages)

- ability to use both templating and overlaying to "shape" configuration data quickly and reliably.

kbld:

- adds image references to make deploys more deterministic

kapp:

- deals with ordering of CRDs, namespaces and other resources in one go

- works with resources in bulk (tracks them automatically via labels, prunes them when they are no longer needed etc.)

- waits for common resources to complete deploy

- does not store any state except one configmap with generated label (no complex failure scenarios).

someone asked me to wrap get-kapp.io (one of my projects) with terraform so that they can pass down infra settings to their k8s config. i ended up writing https://githubc.com/k14s/terraform-provider-k14s (example: https://github.com/k14s/terraform-provider-k14s/tree/master/...). i havent been asked about case where app info is needed by terraform, but may be thats something to add...

https://get-ytt.io (i am one of the authors), one of the tools that inspired yglu, has been designed from ground up with an eye towards security even though its turing complete.

it is based on starlark which is a runtime that does not have facilities to do networking, fs access, etc. building on that, ytt does not allow/provide any kind of non-deterministic operations (access to time, disk, network, random, etc). it expects user to specify explicitly which files to load via -f flag (ie ytt template cannot load random template from fs).

there are several computation attacks that are currently possible (eg infinite loop) but that could be easily addressed thru global timeout for example.

check out interactive playground at https://get-ytt.io

> I like the idea of having a state and resolving differences between desired and actual state, but not enough to rewrite everything in HCL.

i authored https://get-kapp.io based on my previous experiences managing iaas resources as cattle. it strongly revolves around idea of managing chnageset between actual and desired configuration.

it was also built as a tool that solely focuses on deployment to k8s, leaving configuration building (ie templating) to other tools.

take a look at it, would love to hear what you think.

on a side note, someone recently asked me to also wrap kapp as a terraform provider: https://github.com/k14s/terraform-provider-k14s

just use a wrong type (bool lets say), and check out go compilation error. it will tell you what type it wants.

agreed, text templating of yaml (or any structured content) does not make sense. too much context (actual config structure) is lost if plain text is used.

i've collaborated on ytt (https://get-ytt.io) - yaml templating tool. it works directly with yaml structure to bind templating directives. for example setting a value is associated with a specific yaml node so that you dont have to do any manual indenting etc. like you would with plain text templating. defining functions that return yaml structures becomes very easy as well. common problems such as improperly escaped values are gone.

i'm also experimenting with a "strict" mode [1] that raises error for questionable yaml features, for example, using NO to mean false.

i think that yaml is here to stay (at least for some time) and it's worth investing in making tools that make dealing with yaml and its common uses (templating) easier.

[1] https://github.com/k14s/ytt/blob/master/docs/strict.md

i'll throw in one of my projects as a contender: ytt - YAML templating tool - https://get-ytt.io (check out live playground!).

it works with yaml structures (hence avoids text templating problems) and uses familiar python-like language, starlark, making quite easy to get started. it makes use of yaml comments to assign metadata/templating directives to yaml nodes, so it looks something like this:

  #@ load("@ytt:data", "data")

  #@ def labels():
  app: echo
  org: test
  #@ end

  kind: Pod
  apiVersion: v1
  metadata:
    name: echo-app
    labels: #@ labels()
  spec:
    containers:
    #@ for/end echo in data.values.echos:
    - name: #@ echo.name
      image: hashicorp/http-echo
      args:
      - #@ "-text=" + echo.text

it doesn't include type checking, however, it does have a system to "overlay" structures on top of each other via overlay feature -- https://github.com/k14s/ytt/blob/master/docs/lang-ref-ytt-ov.... merge/replace/remove operations expect to find one node by default so map key typos or wrong structural nesting problems are caught easily in common cases.

> ...but the goal is that you should choose whatever templating language you already use for your website

most common templating languages (especially textual templating languages like go's text/template lib) are not great at templating structures (yaml, json, etc.). you quickly run into indentation, escaping and similar problems.

> I disagree with the way kustomize and jsonnet approach this, learning a new templating language is almost never the right solution.

i've recently open sourced a tool ytt (https://get-ytt.io) that is yaml structure aware, to avoid problems mentioned above. i do agree that learning new tool/language is an overhead but it's been interesting to try to build a tool on top of common tech, yaml and python-like language (starlark), to ease the learning curve.

good to see that your tool allows integration with other templating tools.

i actually open sourced a tool kbld (https://get-kbld.io) a couple of weeks ago that includes export/import images functionality (keeps same digests across registries): https://github.com/k14s/kbld/blob/master/docs/packaging.md. didn't think of it as a image backup tool till i read your comment.