Hacker News Reader

Comment by toprerules | Hacker News Reader

The best config system I've ever seen used plain old Python to generate static configs. Everyone knows Python. Python is easy to do data munging in, as demonstrated by it's popularity as the #1 data science tool. There's boundless libraries to make Python more functional, use stricter typing, or reduce the amount of side effects it can cause. Even Starlark is just a dialect of Python.

You can spend decades building a complicated configuration language, use a bespoke functional language as Mill does, but if you're a single company that can enforce code quality and just wants to get the job done, I feel like everything else is just unnecessary and over-engineered to scratch some academic itch for a "better system" that enforces "purity" at the cost of velocity.

I also think that now that LLMs are on the rage, how much context do you think they have for bespoke config language vs Scala vs Python? I think we know the answer to that one.

View on HN

Python is a terrible choice for that sort of thing. Who really wants to have to set up a venv and deal with pip nonsense just to write a config file? Hell even installing Python is sometimes difficult.

Stick to the standard library as of an oldish version of python (3.6?) and it's pretty much zero-install zero-config.

On a Mac, Python has always been a challenge.

Up until recently Apple only included Python2 and so developers used Homebrew to install Python3. Now it’s very common to find two versions of Python3 installed on a Mac developer’s laptop that conflicts with each other.

You really want to be using virtualenv.

Hence the advice to stick to the standard library - because then it doesn't matter all that much. I'm not sure what the full set of environments I've tested my Python 3 scripts actually is, but they've run OK on all the various cloud CI systems I've tried, plus my laptops, VMs, and work PCs.

All a Mac user has needed to do was install from https://www.python.org/downloads/ and then run python3 in the shell. Even if you use MacPorts or brew or conda or whatever, there's a distinct command to run Python 3 instead of 2.

I get that Python's package manager situation is terrible, but like the other user said, you only need built-in packages to spit out a config json or whatever.

So you would then you end up with three Python3 installations.

And if you install from the website it doesn’t override the path. So will still be using the Apple or Homebrew one.

If you install it 3 times then yeah, but even then, all 3 of them will still work.

But I could've sworn the python.org installer set the PATH. If not, that's kinda annoying.

If only Python had the equivalent of npm.

Thought that was pdm. Never saw it used so far.

You could say the same thing about setting up a JVM...

I definitely would!

JVM doesn't have everything all over the place, JAVA_HOME and PATH suffices.

It doesn't require reaching out to tons of C libraries when performance is called for, and there are at least two free beer options to AOT compile to native code, if required.

You don’t need any of that now that we have uv (https://github.com/astral-sh/uv)

Not really. Uv is great but it doesn't really help here. If an app uses Python as its configuration file format then the app is running Python. It's going to do `python3 config.py` or similar. It doesn't know anything about uv.

So you would still need to create a uv project, run `uv sync` and `uv activate` or whatever and then run your app. Not practical.

The only option if you use Python as a config file format is to stick to old features (Python 3.6) and not use any third party libraries. But op was saying third party libraries are one of the benefits of using Python...

This does work well. A team I was on at a past job did exactly this. On Unix the service literally ran `std::system("python config.py >config.json")` on startup.

The problem with this is that the answer to the question "what kind of configuration can I expect?" is "simulate the script and find out."

If the script is written well, and is short, then the parameters that are filled in by the runtime environment are apparent. Over time, though, there is a risk that the script will not remain written well, and it almost certainly won't remain short.

An approach I use is splitting my config tools into 2 stages

Stage 1 creates a "explicit" config that can be exported to plaintext that contains exactly what is going to be created/modified with no abstraction/simplification

Stage 2 applies the "explicit" config

You get to be as clever as you want in stage 1 to avoid excessive copy pasting or not being able to know what your tool is going to do because all you have to go on is some homegrown DSL

You run into the same problem with config DSLs, except now you're dealing with a DSL. Config is almost never going to be static.

True. One advantage I can imagine for a DSL is that it constrains what is possible and optimizes (syntactically) what it's supposed to be for. I think that the author of Nix justified its language that way.

The counterargument is "eventually you'll need every facility provided by a programming language, so just start with a programming language."

I'm not sure how I feel about it. The YAML templating situation in Kubernetes is a [shit show][1]. Then again, I did once cave into the temptation of writing a [lisp-like XML preprocessor][2] to make my configurations less verbose. It doesn't have any access to the environment, though, so it's not a general purpose configuration language, just a shorthand for static XML.

[1]: https://www.davidgoffredo.com/no-string-templates

[2]: https://github.com/dgoffredo/llama

What constraints are needed? I've used DSLs that are almost Python but not quite, I think because they were hermetic and deterministic. Even those ended up being produced dynamically using some higher level config DSL or just regular code. Like once you're doing RPCs, it's general programming language territory (though there are also DSLs that do this, which is cursed).

And yes, I have very bad memories of Kubernetes YAML, also YAML itself.

Scala is hardly some obscure bespoke language. It's a top-20, maybe top-10 programming language, that's been around for 20+ years (and had far fewer breaking changes than Python over that time). Most Python translates directly into Scala, but with the benefit of a proper sound type system and full IDE support. And it's a great language for data munging.

this “top 10” your LLM hallucinating? :)

The claim sounded outlandish, but Scala looks indeed to be around the top 10~20 languages in hiring for instance:

https://www.devjobsscanner.com/blog/top-8-most-demanded-prog...

Scala is only in 0.5% of the scanned job offerings, and is far far behind the major languages in numbers, but I was surprised there's more demand than Rust or even Perl to be honest.

I'm not surprised it's above Rust and Perl, but it's below Dart?! Ouch.

Might as well just list it as Flutter (Dart).

> Everyone knows Python

No they don’t. Just like everyone doesn’t know Cobol, Fortran, Scala etc.

But by having a programming language as your build tool you now make it harder for new people to onboard. As in order to build project they often need to some unique, specific to the language syntax. And in order to find this syntax they look around on Github and because it’s a programming language every project has their own unique, specific to the project approach.

Versus something like Cargo.toml where it’s simple and consistent regardless of which project you look at.

I think it's hard to argue that Cargo.toml is any simpler than Python. Json might be ubiquitous enough for anyone to read and understand, but if Python is foreign than toml is no better.

> No they don’t. Just like everyone doesn’t know Cobol, Fortran, Scala etc.

Sure somebody might not have Python experience, but it's pretty easy to just not hire someone who says they don't know Python and isn't willing to learn for the role. I don't know that you'd filter out many candidates out of any random 100 devs.

I am talking about graduates and others new to programming.

Of course they are willing to learn for the role but making it hard for them in the beginning can forever turn them off a language. That has been a big problem with Scala and Spark.

I've never seen a language used for ancillary purposes be the make or break on hiring for a role, it's always just been expected that you'd pick it up as you go. And IMO, python is the least offensive compared to stuff like Perl, Ruby (for Chef) or whatever the heck Terraform is.

I have onboarded dozens of Data Engineering graduates in using Spark.

In the beginning this was with Scala and every single one struggled with SBT.

Giving developers unlimited flexibility in how they create build files is a bad idea.

Can they use pyspark?

HCL?

> Sure somebody might not have Python experience, but it's pretty easy to just not hire someone who says they don't know Python and isn't willing to learn for the role.

This works just as well for Scala.

There are way more people who know Python than Scala, and it's also an easier language to get started with.

So then they need to know toml (Tom's Obvious Minimal Language)? https://github.com/gtk-rs/examples/blob/master/Cargo.toml I don't know what this file says.

I don't know about Python specifically, but using a language I'm familiar with to generate ninja files (+ any header/environment/&c) for the build has become my go-to way of doing builds in the past 18 months or so.

> I also think that now that LLMs are on the rage, how much context do you think they have for bespoke config language vs Scala vs Python? I think we know the answer to that one.

Nothing against Python, but of all the reasons to choose a technology, whatever is more represented on the dataset of some LLM is the worst reason.

This is a death spiral. There's no hope for the future of this industry if newcomers are thinking like this.

I cared about programming languages when I was a newcomer. Stopped caring about 10 years ago. They're just tools, each with their own gotchas and different design choices I couldn't care less about. Between two tools that both work ok, I will definitely pick whichever one my team and I can learn the easiest, and that includes LLM coverage.

> There's boundless libraries to make Python more functional, use stricter typing, or reduce the amount of side effects it can cause.

What are some examples of a library that can limit or prevent side effects of a piece of python code? I could use one right now.

Ah yes, the age old belief that all software is complex enough that one must first run some other bespoke turing complete program to build every single piece of software.

And of all the languages to pick for this, python, with it's non-hermetic execution environment is bound to bite you in the ass, once your buildscripts start depending on libraries. Oh, you could use poetry to solve the library issue with python, or maybe it'll be setuptools, pip or whatever is the flavour of the month in python packaging.

After fighting with Nix for a sufficiently long time, I think most language specific build tools are not neccesarily the best solution to the problem of automating a build for bit of software written in language X. Complex projects will eventually evolve to depend on multiple languages (unless you're the Linux kernel), at which point the specialized language build tools turn into cumbersome barriers in the build process, where different build tools are not aware of the caching, conventions and configurations of any other tool. As such, in an ideal world, any new language would come with a compiler or bundler that can be supported well by higher level build/packaging tools. And bespoke python scripts ain't that.

Reply on news.ycombinator.com