Jsonnet – The Data Templating Language
jsonnet.org
jsonnet.org
But if I was doing it again now, I would just use https://cuelang.org/ or https://dagger.io Jsonnet is really hard to debug.
I dislike that it's json to transform json, it becomes a rats nest. I had similar experiences with xslt back in the day.
Since jsonnet is side-effect free, types are somewhat less needed. You can instead execute your code, which will generate values and which can report arbitrary runtime errors (all the way to the language server, even)
(Types would still be awesome for writing funcitons, though)
yes! As always, it's mostly a tooling issue. Surely, some aspects of the language like lazy evaluation, make it even harder, but fundamentally there is no reason we cannot significantly improve on the overall experience.
And this is not just about debugging when things go wrong. It's also hard to navigate the codebase and understand where are the templates that you care about. Things can happen at multiple levels (that's the feature!) and so it can be quite hard to figure out which files you need to touch if you want to change something in the output. I'm working on a tool to answer that question: "if I wanted to change this field in the generated out put, what are the places in the input files that contribute to produce this value"
$ ursonnet testdata/child.jsonnet '$.deployment.spec.template.spec.containers[0].resources.limits.cpu'
testdata/common.libsonnet:27
testdata/base.jsonnet:5
testdata/common.libsonnet:23
testdata/common.libsonnet:22
testdata/config.libsonnet:5
Create a framework in a normal language, but one which is very limited by design with regard to what is able to access - Skylark/Starlark is a good example[0], "limited Python" and use that. The rest works the same way, take an input in this language/framework, generate outputs.
You could also try for a templating framework (like jinja2) but then you have 3 syntaxes colliding: the programming language you call the templating engine with (e.g. Python for jinja2) the templating language itself (e.g. jinja2) and then JSON as well.
You are not contributing much here. Could you elaborate what problems they were trying to solve and what other problems were met?
Regardless, the reason is exactly what I mentioned, it’s esoteric, specific, and not backed by a community of knowledge.
Here are some of the things I appreciate about Jsonnet:
- It evals to JSON, so even though the semantics of the language are confusing, it is reasonably easy to eval and iterate on some Jsonnet until it emits what one is expecting - and after that, it's easy to create some validation tests so that regressions don't occur.
- It takes advantage of the fact that JSON is a lowest-common-denominator for many data serialization formats. YAML is technically a superset of JSON, so valid JSON is also valid YAML. Proto3 messages have a canonical JSON representation, so JSON can also adhere to protobuf schemas. This covers most "serialized data structure" use-cases I typically encounter (TOML and HCL are outliers, but many tools that accept those also accept equivalent JSON). This means that with a little bit of build-tool duct-taping, Jsonnet can be used to generate configurations for a wide variety of tooling.
- Jsonnet is itself a superset of JSON - so those more willing to write verbose JSON than learn Jsonnet can still write JSON that someone else can import/use elsewhere. Using Jsonnet does not preclude falling back to JSON.
- The tooling works well - installing the Jsonnet VSCode plugin brings in a code formatter that does an excellent job, and rules_jsonnet[0] provides good bazel integration, if that's your thing.
I'm excited about Jsonnet because now as long as other tool authors decide to consume JSON, I can more easily abstract away their verbosity without writing a purpose-built tool (looking at you, Kubernetes) without resorting to text templating (ahem Helm). Jsonnet might just be my "one JSON-generation language to rule them all"!
---
Though if Starlark is your thing, do checkout out skycfg[1]
i hate working with SBT. Makes maven pom.xml less cruel in comparision.
[1]: https://jsonnet-libs.github.io/jsonnet-training-course/
> Dhall is a programmable configuration language that you can think of as: JSON + functions + types + imports
It is more about schemas and reliably merging documents as far as I can tell. But weirdly given its focus on schemas I couldn't find any way for a document to link to a schema in-band!
Not really sure what you mean by "documents" though.
In cue there's no real distinction between "schemas" and "documents". If you say:
value: string
value: "abc"
For Cue you would probably just want to #include a file... but as far as I know you can't do that.
That makes it far less useful for IDEs, linters and so on.
I don't think there's any reason they couldn't add that feature. Just a bit odd that they haven't already.
FWIW, "YAML Typing = None" is for sure wrong, that's what the `!!map` annotations (https://yaml.org/spec/1.1/#tag/syntax) are for, and a very common RCE vector for dynamic languages since they can cause execution when the parser instantiates the types
I'm still tinkering with how to organize things. I don't think I've fully grasped the purpose of mixins yet.
It's a massive learning curve but it has become really powerful and flexible.
I started trying to use Tanka but found the mental model a bit strange and struggled passing parameters into environments, so I tried Kapitan instead and that has been much more productive.
To clean up the diff a bit I recommend using: https://github.com/sh0rez/kubectl-neat-diff
Hope it’s helpful!
We're using exactly that diff-on-PR/apply-on-merge approach, checking the generated manifests in for easier reviews, but I found just using jsonnet -m made it difficult to DRY things out (e.g. set some URL values differently per environment) so I'd be curious to hear how you approach that?
Would you wrap the customised kube-thanos/prometheus-operator "calls" in another library?
There is also the Dhall and Nickel languages providing features alike
- Policy spec is expressed in protobuf format
- GRPC gateway plugin is used to generate Swagger (OpenAPI v2) spec from proto files
- Jsonnet bindings are generated from Swagger spec.
- Blueprints are implemented using Jsonnet bindings. The users generate policies from blueprints by providing configuration in yaml (using aperturectl CLI) or via jsonnet mixin.
Relevant links:
- Aperture Blueprints (Jsonnet): https://github.com/fluxninja/aperture/tree/main/blueprints
- Generated doc example: https://docs.fluxninja.com/development/reference/policies/bu...
- CLI for generating policies from blueprints: https://docs.fluxninja.com/development/reference/aperturectl...
- Policy spec: https://docs.fluxninja.com/development/reference/policies/sp...
https://github.com/fluxninja/aperture/blob/main/scripts/json...
Something like Starlark is probably a good option.
Imagine having to take a nested configuration fragment that refers to its own pieces via the absolute self, and trying to plant it in some other configuration, at a different nesting level.
Jsonnet’s internal predecessor /counterpart/inspiration (Google BCL/GCL) has something like this and not including it in Jsonnet is a feature :).
A convention I use with jsonnet is to use locals to "anchor" some important objects in the lexical scope. My rule is to name the locals with exactly the same name as the field whose object they refer to. Example:
{
deployment: {
local deployment = self,
spec+: {
replicas: 3,
template+: { spec+: {
local spec = self,
containers: [
{
name: 'foo',
env: [
{ name: 'REP', value: std.toString(deployment.spec.replicas) },
{ name: 'SA', value: spec.serviceAccountName },
],
},
],
serviceAccountName: 'foo',
} },
},
},
}
They choose to use it because:
- it allows them to keep things DRY when replicating a configuration artifact across environments
- it is accessible for folks who are not as comfortable with program control flow and other programming constructs. They can start by copying and pasting JSON.
- it provides more advanced control flow and logic features for those who need them
- it provides features for template parameter validation
- it is not that hard to learn
Some things could be improved for sure. Better IDE support, error messages which are not the best. Maybe also a more confident community.
Overall, Jsonnet does seem to hit a pragmatic sweet spot for a bunch of use cases, regardless of the merits of other approaches mentioned in this great discussion!
Could just be config.js. (Or just generate JSON in your app programmatically, it’s a pretty simple well-defined syntax and there’s already a hash-to-json library for whatever platform you’re using)
In my experience, it works, but it doesn't really give me any distinct advantage and instead it gives me some headaches. Maybe for things that look almost like JSON it'd be helpful, but the moment you start dealing with more complex generations you start finding lack of typing, lack of IDE support, lack of easy debugging, etc, fairly problematic. For example, something I really dislike is that due to how the expressions are evaluated, the only way to add debug/trace statements is to use them to "transform" a value you're going to use, if you don't use the result of the trace in the final output, the trace does not appear.
Also, I really dislike the error messages. Again, due to the lazy evaluation design, when you mess up something the error message might appear deep in some unrelated call stack and with a jarring lack of context. Debugging it is a real pain.
In some of the discussions where the community is trying to figure out how to future-proof current dashboard definitions, Grafana Labs has also recommended this Python tool (unofficially I suppose) by Weaveworks, called grafanalib: https://github.com/weaveworks/grafanalib
Thema is mentioned there, as “the presumed successor to grafonnet,” but that hasn’t officially been confirmed and it sounds like bigger changes might be underway. Discussions are also happening elsewhere, like their Slack, but that link is the most complete overview I’ve found.
yes! As always, it's mostly a tooling issue. Surely, some aspects of the language like lazy evaluation, make it even harder, but fundamentally there is no reason we cannot significantly improve on the overall experience.
And this is not just about debugging when things go wrong. It's also hard to navigate the codebase and understand where are the templates that you care about. Things can happen at multiple levels (that's the feature!) and so it can be quite hard to figure out which files you need to touch if you want to change something in the output. I'm working on a tool to answer that question: "if I wanted to change this field in the generated out put, what are the places in the input files that contribute to produce this value"
$ ursonnet testdata/child.jsonnet '$.deployment.spec.template.spec.containers[0].resources.limits.cpu'
testdata/common.libsonnet:27
testdata/base.jsonnet:5
testdata/common.libsonnet:23
testdata/common.libsonnet:22
testdata/config.libsonnet:5
1. Fast to parse with a small engine, good error messages, safe to evaluate.
2. Powerful, can express config with arbitrary logic.
In Conveyor we try an alternative approach. The config is HOCON, which is a superset of JSON syntax designed for human readability/writability/convenience first and foremost, so it's got a very nice and clean feel to it. You can see an example here:
https://github.com/hydraulic-software/github-desktop/blob/co...
It can be parsed with a normal-sized config library and the errors you get are reasonable.
But then what if you hit the limits of what it can express? We added support for "hashbang includes":
include "#!script.js"
Why. It’s unclear what’s the syntactic structure of it at a glance. Yes, in stricter languages there’s more of /[()[\]{}"'`,;]/, but also I know for sure what’s a literal, what’s an identifier, what’s a key, what’s a number or a date. The whole structure and ordering is obvious.
Same issue I have with nginx, terraform and other ad-hoc half-languages half-formats half-templates. The worst part is that you have to hack them once a year, but it never “sinks in” for good even if you read the docs.
https://conveyor.hydraulic.dev/7.2/configs/hocon-spec/
You can also convert HOCON to JSON and back. I find it a lot easier to work with configs when you can get rid of superfluous syntax, can use comments, substitutions, include files etc. But you don't have to use those.
Nevermind, https://jsonnet.org/ref/stdlib.html
It’s just a new language with a new runtime, for some reason marketed as a json templater. Literally any language with “json” module is at least equivalent to it, but more familiar to developers.
Or make an application run on a normal config.
* Accessing a missing field is totally fine in JavaScript and returns undefined; this is an error in Jsonnet.
* JSonnet guarantees no side-effects, not true of JS.
* When you run a jsonnet file, you have a consistent CLI that handles things like external vars, whereas you'd have to write your own wrapper to do this in JS.
* JSonnet lets you export to multiple different config formats, whereas JS only support JSON really. Again, you'd have to write a wrapper to do this.
* JavaScript would let you return any valid JS object, your wrapper would need to validate.
Like yes, you could write your own JS library to mimic a lot of these behaviors, but the constraints are what make the tool useful. I do wish it had better typing though, so you could more easily inspect the structure of the resulting object.
I also don’t think “object matches type description” is a niche that JS is lacking, there’s tons of libraries out there to assert that.
You’re just trading footguns for different footguns IMO.
2. can be fixed with Deno - https://deno.land/manual@v1.31.3/basics/permissions
3, 4, 5. Why not have that wrapper to solve those? (AFAIK you do need the wrappers for jsonnet too)
Jsonnet's stated aim is to be a superset of JSON rather than something similar to JavaScript.
Having used Jsonnet for a while, it's nice in that it makes it relatively easy to take existing JSON and incrementally turn it into templates. You wind up with the ability to create some nice abstractions for elements of JSON. The close coupling between the syntax of the emitted output and the Jsonnet script itself makes it far easier to write a correct template than when using something like gotmpl to create Yaml in a Helm chart.
Whether or not this is worth another language is a judgement call, but it's not that hard to learn or work with, so I've tended to find Jsonnet a nice tool to have around.
We wanted to avoid having to inject a more complex piece to the puzzle. It worked okay at first, but I knew from the get-go that wasn't going to hold for very long and eventually there was a Python app that took the YAML config and the Jsonnet+defaults and mashed them together before sending the info to TF + Spinnaker.
But yes, I strive to keep the "one file, one target, import whatever you need but explicitly" as much as possible.
I'm pouring some more time into the project and trying to implement some ideas I had for a long time but never managed to get them out. For example "Flags From Files" (https://github.com/kubecfg/kubecfg/blob/flagspec/docs/rfcs/r...) or "Caching + optional vendoring of immutable external deps".
Change my mind. :-P
Look at helm as an example: https://helm.sh/docs/chart_template_guide/function_list/ is some of them, https://helm.sh/docs/chart_template_guide/accessing_files/#p... are some others, but they also glued in some version of https://masterminds.github.io/sprig/ So, short of (a) knowing that's the case (b) having 3+ bookmarks in your favorite browser to refer to those reference pages, how would anyone know what pipelines are available?
Separately, I dooooo nooooooot understand why every joker has to invent their own new thing when we have like 50 or so templating languages already. Golang may be an outlier in that competition due to the Google Promotion Packet Effect(tm) but how they came up with `{{ range }}{{ end }}` as sane syntax is some true facepalm, to say nothing of the same landmine that ansible stepped on by not switching jinja2's default characters: `{{` is not _yaml safe_
If you use go templates directly- it's very obvious what functions are available, there's the default ones from the Go template package, and there's whatever you pass explicitly. The go template package is very small and well documented.
What I love about Go templates, is that I can pass any function I want, using good old Go code (meaning I can use SDKs and whatever else). This is really powerful- we're able to have functions that go out to APIs and return data and it's very easy to manage
Well, that's a tautology when having any contact with golang templates outside of quite literally just importing "text/template" and good luck, as your subsequent comment somehow says and yet misses the point. I draw one's attention to the sibling comment of yet another golang templating gizmo that injects its own cutesy functions with completely random naming into the evaluation namespace