Jsonnet – A data templating language
jsonnet.org
jsonnet.org
It's GCL but with a strong type system and termination guarantees.
I personally like the syntax a bit more than jsonnet. It's a bit less noisy.
I'm also a bit biased because I work on https://github.com/dhall-lang/dhall-kubernetes . We're currently working on a 2.0 release so the current state of the repo is a bit in turmoil.
Programmable configuration language like dhall an jsonet are very powerful and I like them way more than for example helm charts, where templating happens on the string level instead of at the data level.
Alternatively nix expression is also close enough: https://nixos.org/nix/manual/#chap-writing-nix-expressions
Jsonnet is a functional lazy evaluated language that supports powerful referential concepts. Unfortunately the website sucks at getting this across.
JS on the other hand is an imperative programming language. If you say "use JS" you may as well say "use Python", the 2 are equivalent in this context and neither are in the same class as Jsonnet in terms of robustness and simplicity.
I didn't get it until I used it but after using it I don't think I would go back to test templating when it comes to dealing with JSON/things that are equivalent. i.e I would use Jsonnet to create JSON to convert to YAML rather than templating YAML directly. Same goes for generating objects directly in JS/Python/jq - yes it can be done, yes sometimes you can write a nice DSL but in most cases Jsonnet is going to allow you to write said DSL faster using functions and it's going to be simpler and less prone to errors.
We ended making the effort to simplify config of all our app’s components to just read from one large yaml file. Far happier with this solution.
It was discussed on HN though I don't have the link handy.
But sure, if you are okay with a single yaml file, that's certainly simpler.
Let's go through the list of things that the author says Javascript doesn't have:
2. Lexical block scope
JS has lexical function scope, so it would trivial to convert a "lexical javascript" to a "function lexical javascript." I think it's unfortunate that JS doesn't have lexical block scope, but it's trivial to implement through nested function definitions, so I don't think it's a real problem.
3. TCO
This isn't a feature of a language, it's the feature of a compiler. It just so happens that the Scheme spec requires the compiler to have TCO. But if the Scheme spec didn't mention this, would it make a Scheme something different?
4. Continuations
Continuations are very rare in Scheme, or any language for that matter. I think someone would be hard-pressed to argue that it's fundamental to Scheme.
6. S-expressions
There are many extensions to JavaScript, (JSGEN in Allegro, and various other libraries in SBCL), that allows one to write Javascript in S-expression syntax. So maybe while Javascript doesn't use S-expressions, one can very trivially transpile S-exp JS to normal JS.
8. Macros (he probably should have said hygenic macros)
It's pretty easy to implement a hygenic macro system on top of Javascript. In fact, people have already done so, one example is sweet.js.
9. Distaste for mutation
I'm not sure that distaste for something can be part of a language spec. But it's a non-argument: if all JS programmers suddenly had a "distaste for mutation" would it be anymore or less like Scheme? I don't think it matters at all. This point is about culture, not language spec.
So, in conclusion, from the above points, one could trivially write a Javascript to Scheme compiler that would have all the features of Scheme, except maybe call/cc. If this is what you define as Scheme, sure, fine, you win: Javascript isn't Scheme. But for the rest of us, it doesn't really matter and JS is damn close to Scheme from a practical perspective.
Moreover, it wouldn't even be that hard to add call/cc to JS! So I don't think that or any of the author's points are very valid.
And continuations are implemented by convention in nodeJS (through callbacks)
It even is specified as including TCO, a classic scheme feature, but so far it is implemented only in Safari.
Non mutation also seems to be a trend in web development, again thanks to `const`. The only things that make JavaScript not scheme, then, are syntax and the lack of macros. (Which is kind of a result of the more complex syntax)
In this case, JavaScript's syntax is very similar to Jsonnet. Or at least what is shown in the example can be accomplished in a very similar way. (maybe except anchors, which can be replaced by variables)
For people who doesn't know, GCL (Generic/Google Config Language) is a language that uses dynamic scope. You can define a `template` object with a variable defined as `external`. This way you can create similar objects by providing a concrete value for those `external`s at the time of instantiating.
The following paper was referenced in Google's borg paper and gives a good overview of the syntax.
But wait the product have multiple regions. For each region we will depend on regionalized database etc to improve reliability. We need to have some sort of "mixin" so that certain command line arguments can be consistently applied to all jobs in that region.
And then SWE want to rollout a new feature, as the first step they want to roll it out in one data center first then move to full prod later. Now you need to have a data center level overrides.
Can all of these be done in a generic programming language? Yes sure, but why not use a language that intentionally limit what you can do, is almost side effect free?
For example, here's a config for NER with ELMo:
https://github.com/allenai/allennlp/blob/master/training_con...
I guess the only really jsonnet-specific thing here is std.extVar() call, but generally speaking it's just a more flexible JSON file that doesn't cost much more to use (brainpower or physical resources). I use it to mix and match configs while debugging sometimes as you can do things like if statements and adding dicts together.
It's also nice for modularization, you can do things like having a function that returns a dict, so if your config needs two slightly different datareaders you can just pull out all the common parts and call the function twice instead of having two copies of the entire datareader config (with only minor differences).
Tools like Zapier, Customer.io's webhooks, etc. allow integration with arbitrary APIs and have become a standard part of many marketing/sales stacks. These apps use things like Jinja, Mustache, or Liquid which are great for text but not JSON templates.
Curly braces as primary delimiters, not allowing trailing commas on final elements, and other features of JSON make generating it a real hassle.
Jsonnet is too complex to be that tool, but I'm glad people are working in the space.
{
"hello": {"{{place}}":{{{#items}}"{{thing}}":{{n}},{{/items}}}},
"urls": [{{#items}}"{{url}}",{{/items}}]
}
Syntax highlighting, linters, proper indentation, etc. all help...but when talking about non-programmers who aren't used to this it's an incredible hurdle. People are also lazy and error prone, so you can't rely on indentation to make things clear, especially since it's not always clear how you should indent nesting of the template vs nesting of the json datastructure you're trying to build.
Most modern templating languages have a config option for changing the delimeter. Setting it to something like (( or << might help differentiate things a bit, but then you're throwing away easy use of examples, documentation, stack overflow questions, and so much more that already exists for that language. That documentation and copy-pastable community help is not something to throw away lightly if you're designing a tool for this audience.
It's also worth pointing out that the above template generates subtly incorrect json with trailing commas. In pure mustache it's not possible to get rid of that trailing comma. You have to either change your data (add a "last":true key to the last element of any arrays), or have some kind of post-processing step.
All of the existing solutions are workable, they just aren't ideal.
If I had to do it all over again I'd probably use the config option and change them to something like '(.', '.)' that doesn't naturally occur in the output language.
There's already a relatively popular tool for JSON processing called jq, whose name in some languages already reads like "jk".
One thought I had though is that I wish they didn't use the % operator for string interpolation though and rather the more recent f-string syntax. I wonder whether it will be possible to add that or if there is a reason to prefer one over the other in this context?
I wonder if folks had had experience using it for kubernetes configs like https://jsonnet.org/articles/kubernetes.html
This is in fact how I manage Kubernetes manifests.
I use this package: https://www.npmjs.com/package/kubernetes-models
Then I write normal TypeScript code to create Kubernetes objects/resources.
Then I render it to JSON and feed to kubectl.
The beauty of this solution is that it’s OOP and DRY.
He has a TS script that outputs JSON, which he is using as config.
He might trust his code, but I was highlighting reasons why this is not true everywhere, and indeed the ability to evaluate untrusted code can open up previously unavailable avenues. For example, storing server-side Jsonnet that can be safely evaluated by a multitenant backend.
I read everything and I still don’t understand why.
A bunch more exist for things like Prometheus, consul, etc, vault, cluster, ceph, memcached etc
We’ve also built a super simple package manager for this called jsonnet-bundler (jb for short).
And of course we use jsonnet (with the ksonnet lib) at Grafana Labs for all our Kubernetes jobs.
Do I really want imperative code in my config files? And if so, why not just use vanilla JS?
I don't think it would be very pretty since it's not designed for that use case, but the jq language is certainly powerful enough to allow it.
From reading the website I see it actually was inspired by gcl/bcl for kubecfg, so hopefully they used some lessons learned!
jsonnet /dev/stdin <<< '{x: 1, y: self.x} + {x: 10}'
{"x":10,"y":10}
Aren't they supposed to be predefined values?
If you have many different components, you really want to factor out common things into templates (like build, deploy), with the option to disable some stages (some components might not have end-to-end tests, for example), plug in some values etc.
I've long been on the lookout for a template system for data structures for this use case, so I'll take a look at some of the suggestions in this thread.
For example, embedding a bash script in a templating language. This is such an incredibly crap idea I don't know how it keeps getting perpetuated. Use of this pattern works as follows: user A writes the script, tests it, shoves it into template. user B modifies the template, commits it, it later breaks silently because it was not tested, because it's in a template language, so you can't lint it and it has no tests written for it (because how did you know it would even get generated based on your input/function, let alone all the places it would be used and how, because templates). user C finds the hidden breakage after tearing out hair, removes the script from template, and writes it as a normal script that takes options and writes a test for good measure. (If you try to use normal dev patterns around the templates, like generate-template-then-commit-to-vcs, relying on and testing the generated versions, your tests have to be reworked whenever you regenerate, and later the generator gradually begins to break, so you just modify the copy in VCS rather than using the template)
A separate problem is that templating languages that give you advanced functionality lead you to want to use that functionality, and the more you do, the harder it is to maintain over time. It may not be readily apparent that trying to make Ansible configs work declaratively requires an annoying amount of complexity, or writing entire applications that cannot be easily reused and calling them "modules" or "plugins". (Whatever you are trying to template/generate, the only way to know for sure you're doing it validly is to have a function that actually parses and outputs the format according to a spec, because just spitting out a template and then externally validating it becomes very difficult to debug, because you don't know why the template is spitting out the output the way it is. So your templating language eventually becomes a compiler)
Since this is a new language, you also have to re-implement everything you normally have for a language, like a plug-in for your IDE, linter, debugger, code analyzer, port the standard libraries to different systems, etc. And of course all new devs have to learn this thing and all its quirks before they can start doing work. An entire ecosystem eventually needs to exist to support its continued use, and if you have problems, you hope this ecosystem continues to flourish so you can get support for this unsupported language tool thing, or you'll be hiring internal devs to do it.
Maybe the problem is these start out as tools for developers, and only when they get used for operations do the anti-patterns surface. I've gone through it several times and it always ends up a PITA.