A new configuration language. Yes. That's exactly what we need.
Data serialization languages (like JSON, INI, XML, etc) lack the power to describe large configurations. There's no way to define an abstraction and then use it in multiple places. There's no way to constrain what is considered correct. You end up with tens of thousands of lines of very, very repetitive structures that are very fragile and hard to change.
General purpose languages are also bad for writing configurations. Yes, they're very powerful. That's not a feature when it comes to configuration. If you have a program that emits a configuration, the only thing you can do with it is run it, then inspect the resulting configuration. You can't inspect or transform the program at the level of the configuration semantics. You can't ensure that the configuration will have specific properties. You can't even ensure that the program will, in fact, emit a configuration.
So yeah, we need new configuration languages that lie in between data serialization languages and general purpose computing languages. We're starting to see them. HCL is awful, but it was an attempt at solving this problem and a move in the right direction. Jsonnet is maybe better better? I dunno, I've never tried it. Dhal is interesting, though difficult for non-Haskellers to approach. CUE is also interesting. Bravo!
We have all we need. If you want a full featured language, use that. Beyond that, we have JSON, YAML, INI and more. If you want something more complicated you can create your own DSL for your app.
If you want schemas and validation, use XML!
- 'dumb data' (json, yaml, ...) is machine-, but not human-friendly. it's often tedious to write/read, and you can't abstract common parts out
- DSLs are something you have to write, debug and maintain yourself. is the bug in your config or is it in your DSL's implementation? who knows!
- full featured languages require a full blown interpreter and aren't tooling-friendly. as an example: with a Python package, it's not really possible to statically determine the dependencies , because its setup.py can declare anything it pleases depending on, say, the time of day. also you can't really run untrusted configs because they might launch some missiles
---
there's a decent middle ground – write a program that generates a 'dumb data' config. you write a small amount of code (friendly for humans) and run it to get a static, easy-to-process config (friendly for machines). however in practice (in most languages) the program won't be pretty – probably about as easy to read/write as an implementation of a macro that directly manipulates ASTs (i.e. not very). this can sometimes be ameliorated with some EDSL trickery, but that brings back all the problems DSLs have
and so generating configs is what projects like Dhall/Cue aim to improve. i'd say they're aiming to be something like the regex of config generation - do a limited amount of common&useful things, and make them easy to express.
If Cue is basically a better YAML with built-in schemas, that sounds pretty good.
And that seemed to be the most mature TOML parser that I could find for Python, so I decided TOML isn’t likely in my future.
I’ve used JSON for configs, which is not great, but at least readable. The biggest problem is lack of comments, but I added a quick hack to ignore any dictionary key starting with an octothorpe. Not ideal, but is actually handy because it makes it easy to comment out keys temporarily as well as add arbitrary commentary.
Also, the Python standard library JSON parser is very flexible and yields precise exceptions. I was able to turn those parsing exceptions into meaningful error messages with precise line and column numbers. Which came as a bit of a shock to some of my users, because a different tool written before my time but used by the same people had exactly one error message for mispelled YAML: segfault.
I never really understood this. This seems to be an oft repeated truism from mid 2000s with little backing it up.
The only awesome thing that JSON did, was lose type information as well. In fact, the only thing that I can see that JSON brought to the table was easier editing by those who didn't have IDEs, at the expense of losing type information.
I can’t just load the file and run it through a parser and get an easily accessible object structure back, I’ll need to navigate the document with DOM or XPath.
JSON with comments (aka JSONC) is in my opinion the best format in wide use today. It has structure and types, but not too many, and not a lot of magic, like YAML has, but for the most complex cases, JSON(C) falls short with its lack of extensibility and inheritance.
Like a simple configuration access and management (SCAM) protocol.
Yes, yes I think we’re on to something here.
Go ahead and write your configuration as a fully Turing complete sub-application in whatever language you like. It will do everything you could possibly want, and in a few years it'll grow so complex and hairy that it will need its own fully Turing complete sub-application to configure it, lather, rinse, repeat. If you're really clever, every configuration layer will be written in a separate language with its own dependency tree, test framework, and toolchain.
Personally, I prefer not having to recompile just to change some variables and settings. I'm fine with INI or JSON (although I prefer Lua tables) when they're appropriate. The problem is not that configuration languages are a bad idea, the problem is interminable Turing creep and developers wanting every aspect of their applications to be as flexible and powerful as possible.
The premise of having a separate config language is fine - somewhere, somehow, inevitably, you're going to need a read-only data store for globals and references to system settings. You can hardcode all of those variables in your application or put them elsewhere.
I'm genuinely intrigued to see a non-trivial example of this, a configuration in use by an organisation with complex needs, describing its applications and its cloud infrastructure, while avoiding the pitfalls you describe that bedevil the use of Turing complete languages.