Hacker News Reader

Comment by Eduard | Hacker News Reader

Programs should not only report where the configuration files are located, but also report how configuration options contained therein are read in:

Some programs use a "last occurrence wins" approach, while other programs (e.g. sshd) use a "first occurrence wins" approach.

Buried in https://man.freebsd.org/cgi/man.cgi?sshd_config(5) and awkwardly expressed:

"Unless noted otherwise, for each keyword, the first obtained value will be used."

That's why e.g. Debian's /etc/ssh/sshd_config ( https://salsa.debian.org/ssh-team/openssh/-/blob/debian/1%25... ) starts with

  Include /etc/ssh/sshd_config.d/*.conf

, so that custom configurations in that directory win over the distribution-provided defaults set up after this Include directive inside /etc/ssh/sshd_config.

View on HN

And yet another gotcha with configuration file loading:

Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.

E.g. from systemd.unit manpage ( https://www.freedesktop.org/software/systemd/man/systemd.uni... ):

>...

> In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc/ take precedence over those in /run/ which in turn take precedence over those in /usr/lib/. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.

> ...

And this is just an excerpt which doesn't cover the details. Read the manpage for all the hilarity.

>Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.

>Read the manpage for all the hilarity.

If you find it hilarious it's because you don't understand the reason. /usr belongs to the distribution and /etc belongs to the user. The distribution doesn't touch /etc and the user doesn't touch /usr.

Upstream ships a unit with its own opinion of the defaults. That unit goes in /usr.

Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.

User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.

Some units are created at runtime on demand and thus ephemeral, like mount units generated by generators. These go in /run. These are functionally "system-provided units" because users must be able to override them, so they're below /etc in the hierarchy.

The "defaults" and "overrides" we're talking about are both at the level of the individual setting as well as at the level of the whole file. If a user wants one of the settings to be different, they can add a new 99-foo.conf file that unsets / changes it. If they want a whole file to be different, they can add a new file with the same name that is empty / has different values for its settings.

BTW `systemctl cat whatever.unit` will print all the files that systemd considered in the order that it considered them.

>And this is just an excerpt which doesn't cover the details.

The only detail your excerpt doesn't include is the consideration of `service.d/`, `socket.d/` etc directories which provide overrides for all units of that type. It's just one point, not commonly used, and not that big of a deal either.

> If you find it hilarious it's because you don't understand the reason.

I think it's hilarious that you think users are supposed to understand "the reason" at this level of detail.

When we make a thing, we make choices about who it's for. Sometimes those choices are explicit. But this is a good example of an implicit choice. Clearly it grew into this shape for reasons, but those reasons are about the people doing the making, not the people using it.

Even if we grant that all of this hierarchy is necessary, it still imposes a large burden on the person trying to figure out "why is X doing Y?" And that might be fine if the makers said, "Well this is going to be hard, so what can we do to make it easy?"

But denying that users will reasonably find this hilarious (or frustrating or impossible) is just digging the hole deeper. Telling people that their understandable reactions are wrong doesn't change their reactions, it just gets them to stop mentioning the problems around you.

> it still imposes a large burden on the person trying to figure out "why is X doing Y?"

Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.

Personally I have found the systemd hierarchy to be very useful. It allows me to create overrides for certain properties in a Unit without having to worry about keeping the rest in sync with upstream, or my package manager overwriting my configuration. And it allows me to have different independent packages (or chef recipes or ansible playbooks) that create different override files for the same unit without stepping on each others toes.

Yes it is a little complicated, but for most end users, you don't really need to worry about most of the complexity, you just need to know where you put your overrides. And for distributors and power users, that complexity serves a useful purpose.

> Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.

Sure, it could be that systemd does have amazing tooling for helping users with the burdens they have created. Or, given the very mixed feelings about systemd, maybe they've tried to be user focused but not hit the mark. I don't know enough to say.

But my point is that either way responses of the form "it's so easy, you just have to remember [9 paragraphs of detail users don't care about]" are part of the problem, not the solution.

Systemd didn't create this problem. But it solves the issue of distributions shipping config files as part of packages and then on every package upgrade having to reconcile between the distribution's config and your modifications. Now with the drop in system you don't need to do that.

I am not an expert but I think in general systemd has a lot of complexity but it's to handle existing issues in a better way. Some of the older init systems might be simpler to describe or get started but lead to more confusing situations in the long run.

"Make it as easy as possible, but not simpler". I think this applies perfectly here. Systemd has very complicated requirements due to its very unique position in a Linux distribution. And as a (power) end-user of it, you basically need you basically needs to touch only /etc

> Systemd has very complicated requirements due to its very unique position in a Linux distribution.

But are all those requirements actually necessary for the job it's doing? Or have its developers imposed artificial constraints through their choices in designing systemd that make it more complex than it could otherwise be?

The combination of distributions existing and believing they “own” parts of the system, and users needing to override settings means it is as simple as possible, and no simpler.

Put more generically, I think we could add: “unless you build it all yourself, you’re going to have to work with other peoples decisions.”

A distribution 'owning' a directory means that the package manager is allowed to change files in there, and allowed to assume nothing major was changed from what it installed.

A distribution that doesn't own any directories like this effectively can't make any changes. It's like writing a class without private fields.

That's because some features are targeted at different audiences. End users want simplicity and ease if use. Admin also want those, but they are secondary to determinism, consistency, and ease of tracking changes.

I find the best way to think about systemd is as the result of some people that probably knew the most about init systems remaking init to be consistent, documented, and handle a bunch of new features that different newer init systems had used to good effect.

Then, because regardless of how much those people knew the vast mofority of information about needs was sequestered in 1000 different teams and projects, they had to add a bunch of new features. Repeat that 100 times, and you get what we have today, which is a system that mostly works (and works well if you actually adopt their preferred patterns), but one that also has the craziest and hardest to remember directive names I have ever seen, where slight nuance about how they function means entirely different params.

In a perfect world it would be rewritten with the same features but with all the knowledge of what features are actually needed known and consistently implemented with sane naming from the beginning, but nobody wants to go through that again.

That last bit is especially what interests me about systemd. From the way you describe it, it sounds like they made it mainly to please themselves, and were then forced to accommodate the actual users and actual users. But that they did so grudgingly and partially, again favoring their interests.

Assuming that's how the history went, fine. Everything has a history, and we have to deal with the now.

But my point is that we are not going to deal well with the now, let alone the future, unless we recognize the impact the history has on people. So if people defensively shout "that's not funny" and imply users are wrong for being bothered by the mess, then we're on track to build another mess. We can't do better until we are honest about why it didn't turn out as well as we would have liked.

> it sounds like they made it mainly to please themselves, and were then forced to accommodate the actual users and actual users

I think it's less that than that there were real advances they were after, which they delivered (Poetterings original essays on what it can and should do and where the ideas came from, such as OS X's LaunchD and some things from Windows, etc) referred to numerous things that every sysadmin I knew at the time that read it was excited if could be delivered (but none of use really appreciated what it meant to deliver them).

The issue is not that they initially built it to their own desired, but that the way SysV init worked with just launching bash scripts to start and control the process that eventually should be running meant that a lot of crazy behavior was put into scripts along the years, because when you can write a program to control your program, sometimes it's easier to just put special behavior into your launch script than to patch the real code in question in some way.

When this crashed against systemd's desire to know what's actually running and control the environment perfectly (which is beneficial, it was always a pain to have a process that started fine from your root user shell but not on startup), and strongly disincentivize programs between systemd and the process in question (which a bash script is) for reasons of process and resource tracking (Linux kernel cgroups were just starting to widely be used beneficially around this time), there started to be a mass influx of feature requests for behavior that used to be handled by dedicated bash scripts previously, but now systemd needed to account for.

Combine that with systemd adding real dependency tracking (as well as start order tracking, which is related but not quite the same, leading to the difference in Wants and Requires), and what we have is a system that's just trying to do something vastly more complicated than what SysV init did previously (and that's before we get to the scope creep which caused them to want to take over some other early boot processes to they could be deterministic or whatever, which is also a PITA for many people).

If you're actually interested in the history if this, I recommend reading "systemd, 10 years later: a historical and technical retrospective" at https://blog.darknedgy.net/technology/2020/05/02/0/. That's where most my info on the details come from. I was around in the beginning and was excited to see systemd start, but I didn't actually use it until far later since most the systems I administer don't get changes like that immediately (and it was in an extremely slow release cycle at that time).

If you read the above you may come away with a different perspective than me, but I will say that systemd is very consistent in its operation and very robust in how it tracks processes and whether there's been a problem and restarts are needed, etc, as long as you actually adopt their preferences (which is to say run everything foreground and let systemd deal with STDIN/STDOUT and environment, and have a good pid to start with). If you do that, writing a new service file is extremely easy. Far more easy than it ever was with SysV init. It's a different paradigm, and sometimes it can be a pain when some process doesn't want to play nice, but mostly it just works once you stop fighting it, which is nice, and cuts down on admin time figuring out problems (especially since it just grabs all STDOUT and STDERR for you so random error messages that don't go to logs aren't lost either).

Thanks for the details. I will check out the link.

I should say that "real advances they were after" is not at all incompatible with "made it mainly to please themselves". The things that pleased them may well have been "real advances"; they often are. The question is the extent to which they were prioritizing the needs, capabilities, and interests of the whole audience being served, not just the things they personally thought important or preferable.

Users are only supposed to know about `/etc`. The rest of the directories are useful for distribution maintainers, people who package software for a distribution (but may not be maintainers), and some developers.

> Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.

> User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.

Larry Wall elegantly solved these problems 38 years ago with the "patch" utility. Debian is a good example of how its use has been standardized to solve these problems.

The patch approach to managing distribution-provided configuration files is superior to the systemd unit approach for at least three reasons:

- Non-experts can read the configuration files from top to bottom in linear order, so the documentation can be inlined by the distribution provider, which means they can add maintainers notes as necessary.

- It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.

- The implementation of all this is factored out and standardized across 1000's of packages, so it only needs to be implemented once and understood once.

Rebasing patches is pretty hard I'd say. Much harder than just explaining someone the logic behind /lib, /etc and /run. I mean, this directories are seperate for a reason. Knowing the idea behind the directory split (a.k.a. Linux Filesystem Hierarchy Standard) is more essential than knowing how to use the patch command line tool. There are more and more people who don't want to use the command line. And that's completely fine.

> - It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.

Anecdata, but still: Every single time the package manager asked me what to do, it was not because of a breaking change, but simply because the base file changed in a way that the patch didn't apply cleanly anymore. That is, I, the user, had to work around limitations of the patch format. If the distro configuration and the user configuration had been separate files, things would have just worked fine.

You don't have to be an expert to read multiple files. If you do need expert assistance with something like that please contact me and we can work something out.

The "expertise" is in finding those files and understanding their hierarchy

and the user doesn't touch /usr.

So it's not hilarious statements like this make it hard to understand the reason? I almost cannot tell whether your whole post is sarcasm, or just showcasing irony.

You mean that the top-level Linux directory are hilariously badly named?

If so, I don't think anybody will disagree.

But if you meant that's a large problem, then well, no it isn't. There are 7 of those badly named ones. It's a matter of reading half a page of text to see what they do, and after 2 or 3 times you read it as reference, you have them memorized.

If would be nice to solve the issue, but it isn't anybody's priority.

Or you'll never memorize them because those 2 or 3 times are many months apart since you're just a user that rarely tweaks those configs

But yeah, ease of use is nobody's priority

Yes, "The user (syadmin) doesn't touch /usr, except /usr/local", where compiled in applications reside with full path so /usr/local/{bin,sbin,share,...}.

If you're deploying an especially big application, you may also prefer /opt/$APPNAME/{bin,etc,usr,...} (A full filesystem tree, basically).

We manage a fleet of systems at work, and I don't remember editing or touching a file under /usr, unless the period I was developing .deb or .rpm packages to deploy on said fleet.

There are many written and unwritten rules in Linux world, and sometimes people disregard or ignore these rules so blatantly, so I can't understand whether the post I'm replying to is a sarcasm or irony.

The rules are very much written down. For instance, here are the rules for /opt:

https://access.redhat.com/documentation/en-us/red_hat_enterp...

There's also a section for /usr.

I’d start here:

• Filesystem Hierarchy Standard <https://www.pathname.com/fhs/>

• For Debian (and derivatives): <https://www.debian.org/doc/debian-policy/ch-opersys.html#fil...>

And, of course, hier(7):

<https://manpages.debian.org/stable/manpages/hier.7.en.html>

I think there is a gap here between those who routinely (exclusively?) used UNIX machines along with 50, 100, 1000 other people all logged in at once (generally pre-Linux era) and those who only ever got to see and enjoy a time when everyone could have a UNIX workstation on their desktop (Linux, Mac, or other x86 flavour, take your pick).

Yeah, I really feel like we're in a "horseless carriage" or "radio with pictures" era for computing. We have incredibly powerful machines available. But the dominant paradigm for managing that power is to slice it up into things that simulate much less powerful processors in software environments designed for a shared 1970s university department minicomputer.

In many ways I love it, of course. It's what I grew up on, and I'm sure I'll have a small Linux cluster in the retirement home, because that's how computers are "supposed to be". But I think the old paradigm has only lasted so long because we have so much computing power to burn.

I'm not sure I understand. When the "old paradigm" was new, computing power was much more limited. They couldn't afford to "burn" any computing power. But it's only stuck around, because now we have more computing power? That doesn't make sense

You might read some Kuhn. Paradigms change when the old one becomes unsupportable. But the vast increase in computing power per dollar means we can waste a lot of it without it being economically uncomfortable.

Well, most of us. A good check on this is a podcast where Dr Kevin Bowers is interviewed: https://www.youtube.com/watch?v=GlzWM5GKQT8

He works for a financial trading firm, where the competitive nature of it means that every processor cycle counts. For him, older paradigms are perplexing and irritating, because they are major barriers to getting the most out of the hardware.

Back in the shared era, the machines were better-suited to individual customization, and were more easily understood by system administrators. Now, they're optimized too... waste more computational resources when idle?

It's both hilarious and unintuitive, but for some people their personal familiarity trumps most other things

X11 clients and servers are now whispering "Hold my beer."

If you want to use /usr for yourself, be prepared to throw the FHS and compability with almost everything away and rebuild your whole userland with something like NixOS/Guix.

It's makes a lot more sense when you realize that /usr doesn't stand for "user", but "user software resources".

/usr stands for ‘user’.

In particular, in our own version of the system, there is a directory "/usr" which contains all user's directories, and which is stored on a relatively large, but slow moving head disk, while the othe files are on the fast but small fixed-head disk. — https://www.bell-labs.com/usr/dmr/www/notes.html

Isn’t that just a backronym? I read recently (here, I think) that on some original specific UNIX system, the user directories were there originally, and then other stuff that belonged elsewhere went in there because the other physical volume (where things like /bin were) ran out of space. Later another dish was added and profiles were moved there, to make /home.

/usr was added when Dennis Ritchie's and Ken Thompson's /home partition was full and they added another disk - or so the story goes.

unix system resources

unlikely semantic revisionism

https://unix.stackexchange.com/questions/103347/pronunciatio...

Nicely linked from one of the answers: https://youtu.be/tc4ROCJYbm0?t=809

Look, mac, I can use google unassisted.

Like I said, revisionism. Also known as a backronym. That's the entire joke.

I can't see how this doesn't make it more hilarious. It gives me "There's no fighting in the war room" vibes.

This is why /usr/local/bin is located in /etc right?

It’s still hilarious. The fact that there’s a need for this hilarity doesn’t negate that. What a mess humanity have created… (Let’s write a suckless OS!)

At least acknowledge how retarded this all sounds.

systemd configuration lookups are lightweights and straightforward compared to ansible variables lookups rules : -- Understanding variable precedence

  Ansible does apply variable precedence, and you might have a use for it. Here is the order of precedence from least to greatest (the last listed variables override all other variables):

        1 command line values (for example, -u my_user, these are not variables)

        2 role defaults (defined in role/defaults/main.yml) 1

        3 inventory file or script group vars 2

        4 inventory group_vars/all 3

        5 playbook group_vars/all 3

        6 inventory group_vars/* 3

        7 playbook group_vars/* 3

        8 inventory file or script host vars 2

        9 inventory host_vars/* 3

        10 playbook host_vars/* 3

        11 host facts / cached set_facts 4

        12 play vars

        13 play vars_prompt

        14 play vars_files

        15 role vars (defined in role/vars/main.yml)

        16 block vars (only for tasks in block)

        17 task vars (only for the task)

        18 include_vars

        19 set_facts / registered vars

        20 role (and include_role) params

        21 include params

        22 extra vars (for example, -e "user=my_user")(always win precedence)

‐‐

that is a RedHat-isim

IIRC, Ansible had this exact variable precedence long before they were acquired by RH.

I get that hating RH is in vogue currently, but let's keep it grounded in reality please.

EDIT: And for the record, these Ansible precedence rules are not a bad thing. Once you are somewhat familiar with them, it allows you to write incredibly flexible and extensible playbooks and roles.

As an Ansible person, I couldn’t agree more. Being able to manage variables at basically any level of precedence is extremely useful.

I am also ever more deeply concerned about IBM’s hell-bent purge of all the good left at Redhat. I had thought it would take them less time to destroy Redhat, but I’m sure they wanted to make sure they spent time “listening” before making all these ecosystem-eroding changes.

Meh, it's not Red Hat. Configuration management tools do this. Chef is similar.

Chef is worse. It's discouraged to use `override` in Chef cookbooks but nothing says that you can't, and introducing spooky action at a distance is just part of your life now.

I love how there are 21 places to look if you're trying to figure out why a CLI parameter is being ignored. Does it at least print a warning or something?

Other way around. Cli params have highest precedence.

According to the comment above, `cmd -u my_special_user` will only use the value "my_special_user" if none of the other 21 places have another configuration for the user, right?

And CLI params override everything, globally.

That doesn't look too bad to me.

".d" directories are a well-established pattern to split a complicated config file into logical units. In the case of something like systemd this is absolutely essential, as you wouldn't want the config for your web server in the same file as the mail server.

/usr/lib is the location where the package manager will drop config files. This is the default, distro-provided configuration. /etc/ is where the administrator can place config files, providing system-specific overrides. /run/ is for files "created at runtime", I am not quite sure what the point of those is.

/run is useful for configuration generation. Sure, you could maybe poke an API and have it do the right thing, but if "everything is a file" then why not put your configuration in a file? That way it's easier to see what's actually happening.

Systemd is also a useful demonstration of how to show people where configuration comes from -- `systemctl status <unit>` will include a list of all the files found during lookup, and `systemctl show <unit>` will give you the resolved unit configuration including all the defaults.

Drop in files (config in .d folders) are a core extension point of systemd and not a symptom of an overly engineered config system. These are how services are meant to be custom tailored to specific machines and even user's needs.

Let's say you need to pass a special flag to sshd on your system, a bad approach would be to go edit the distro shipped sshd.service file. Your change will work until the distro ships an updated file and you forget to hack in your change again.

Instead place a sshd.service drop-in config file in the appropriate place to customize the service as necessary. Your config tweak takes precedence over the distro shipped config, but doesn't live inside it or conflict with upstream changes.

It's a little bit away from the domain of config files, but I think CSS is the most graceful example of handling this as a generalised problem. Style configurations cascade, and the last one read is the one that gets applied. There are overrides to that flow, but you have to very overtly add a piece of syntax to the overriding line like `!important`. Two overlapping overrides revert back to the "last read" approach. The approach is so fundamental to CSS that the entire language is named after it, so there's no ambiguity or forwards compatibility concerns in this regard whatsoever.

I’m sure you know this, but for readers who don’t, it’s much more complex than “last defined wins with a few override options”. Based on “specificity”, rules based on the selector used.

Some examples:

1. #mydiv overrides .mydiv - IDs are more specific than classes

2. .outerdiv .innerdiv overrides .innerdiv - nested selectors are more specific

3. .mydiv overrides div - classes are more specific than elements

And a lot of other rules.

I can’t imagine I’d want this level of complexity in config files.

Nesting is not actually more specific.

Rather specificity is a triplet of (ids sub-selectors, class sub-selectors, element sub-selectors). When attributes conflict, the one whose rule has the more specific selector wins.

The specificities of the rules you wrote out are (1,0,0), (0,1,0), (0,2,0), (0,1,0), (0,1,0), and (0,0,1).

In-line attributes (@style) beat out out-out-of line ones except if they’re !important. Specificity also disambiguates between !important attributes.

Thank you for the clarification, I hadn't realised that.

> I can’t imagine I’d want this level of complexity in config files.

Lots of people don’t want it in their CSS either. It’s generally considered good practice to make everything the same specificity and avoid !important.

> avoid !important

Sure, haven't seen that in a while in well structured code bases.

> make everything the same specificity

How would you achieve same specificity on everything? Only use either ids or classes? No type selectors / pseudo-classes / pseudo-elements / multiple selectors? I can't imagine how that would be feasible and am not aware of anyone recommending this.

With :is and :where it is trivial to trick selectors

:is(selector_1, :where(selector_2))

Has always the specificity of selector_2, and if you choose an impossible selector for selector_1 (the easiest approach is to give multiple ids or use non existent classes/elements) it only select by selector_2

That looks far more complex than just using specificity properly as intended.

There are a large number of companies that require ids as the basis for every selector, by policy. This may not be interesting to you, but I think this is what they meant. The fact they may or may not be familiar to you is incidental.

I'm familiar with policies that generally strive for specificity of (0, 1, 0) like BEM or (1, 0, 0), but I'd be very interested in any policy that rules out pseudo-classes like `:hover`, as prohibiting them would leave you to handle any interactivity you'd get for free from CSS via JavaScript.

This is effectively how Tailwind works. You end up with classes like `hover:text-black`.

Perhaps you mean the same kind of specificity.

Specificity is at the heart of CSS, you're not supposed to design rules with equal specificity

You might want to read on specificity because it’s a huge spanner in your assumptions.

> Specificity is the algorithm used by browsers to determine the CSS declaration that is the most relevant to an element, which in turn, determines the property value to apply to the element. The specificity algorithm calculates the weight of a CSS selector to determine which rule from competing CSS declarations gets applied to an element.

https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity...

Not true at all. There is a very much non-intuitive weighting system applied to CSS selectors which is what ultimately determines which gets applied: https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity

The rules of CSS specificity are a bit more involved than '!important' or not, and I frequently see beginners trip there, but apart from that I agree that the CSS rule set might be worth incorporating.

Fun trick to use if you use a css post processing (less, sass, etc.) is "&&&", it will resolve to a very specific rule which overwrites the other rules

> Two overlapping overrides revert back to the "last read" approach.

IIRC, there's an important (heh) exception to that: user stylesheets. Without the "!important" override on the user stylesheet, the site stylesheet wins; with the "!important" override on the user stylesheet, the user stylesheet wins, even if the site stylesheet also had an "!important" for that rule.

The domain of style and configuration are very, very different. I'm not sure what works okay for displaying a website in different layouts on vastly different unknown devices share a lot of the same concerns of configuring programs on a machine

While at it I’ve made it a practice to expose an internal endpoint for every service to output its config. It has saved me on numerous occasions to catch config related bugs.

Could you explain a little more? I am not sure what exactly you mean by "exposing an internal endpoint" - do you mean that as a secured API endpoint which you can call? Isn't that hard to secure?

Depending on your environment, endpoints might not default to exposed. As an example where I work, we have to explicitly enumerate endpoints to expose (potentially including wildcards, like admin/*), and anything else is only accessible with direct access to the pod.

Presumably you can access it while on a secure VPN but not publicly. You can do this by serving the config route on an internal port.

Indeed, that’s exactly how my team did it.

This is not an endorsement, but Spring Boot has this built in (but not enabled by default):

https://docs.spring.io/spring-boot/docs/current/reference/ht...

I have been leaning towards a different approach. Rather than reading multiple config sources at runtime, read them at build time, and emit a single config file with everything fully resolved. Put that on the filesystem, and have the app read it. You have an easy canonical reference for the configuration, with no need to expose anything. The lifecycle of the config doesn't have to match that off the app binary, so it could (and probably should) have a separate build and release process.

Amazon had a hierarchical configuration system for loading config that could override other config (or add to it, depending on the implementation). The Java framework would provide an endpoint to show you the final resolved config along with which file and line that value came from. Very helpful when trying to figure out why config was different than expected.

At least the inspector in Safari (and maybe other WebKit browsers) does something similar for CSS.

I’m all for it. It’s a pain when writing config systems (no longer just key/value, but key+value+meta), but very helpful. It can be a pain for things like JSON where libraries don’t give you that type of diagnostic information easily, however.

I had the distinct pleasure (?) of owning the C++, Python, and Java implementations of that config for a while and ... yeah, it had some positives, but the fact that it had two orthogonal hierarchies of configuration in addition to implementation-specific feature gaps and nuances made it something of a nightmare in practice.

Agreed, logging in config retrieval code should log all this stuff, requests, where they are filled from, that sort of thing.

The problems with this approach when it's enabled is that log aggregators (e.g splunk) can end up including stuff like passwords which shouldn't be visible to the group able to view those logs, so some care is needed.

I'd rather have some config reserialization capability with provenance (perhaps as an opt-in). A good config reserialization can make itself useful in so many situations, if it has the right features. Like pointing out key-value pairs that don't have any effect, or simply writing out some documentation.

Logging has its merits, like discoverability when you stare at it without suspecting configuration to be the culprit, but I believe that there can and should be more.

Logging keys only should be sufficient for this.

Ant also had first writer wins. I tried to explain this so, so many times to so many people, and only rarely did it stick. On that project I spent way too much time fixing build errors, and ended up rearranging things to make the behavior more obvious rather than less, because without it people would fool themselves into thinking they had it right, because sometimes it would appear to work.

And the thing with developers is that they make lousy scientists. If they want something to work, they’ll take any information that supports that theory and run straight to the repository with it.

The trick is not to fool yourself, and you’re the easiest person to fool.

I would prefer a convention… but first wins is backwards IMO. It really sucks when a bad convention gets adopted, like this first wins, or spaces instead of tabs. ;)

First wins makes some sense from a security pov. You want to load the more "trusted" configuration files first because they must be able to further restrict the less trusted files, but don't want to let the less trusted files override the settings set by the trusted files.

Rejecting multiple definitions is better, unless you care about DoS (I would be pissed if I got locked out of ssh access due to an option being defined twice, for instance).

At least with the spaces/tabs my IDE can automatically convert it for me.

First/last is ambiguous anyway and implementation dependant.

How so?

If there are multiple files in a wildcard pattern, what order are they resolved? I know for classpath loading, for example, it’s in directory order, which is file system dependent, and often based on the order the files were written to the directory and/or the size of the directory inode. Lots of fun figuring out why the wrong version of a class is consistently loaded on one or two servers out of twenty. (Always check your classpaths for duplicates kids.)

> If there are multiple files in a wildcard pattern, what order are they resolved

If you don't care (eg [0]) then do nothing and let the user bang it's head and reinvent the wheel.

If you care - just sort it and mark it as 'platform dependent' or somehow. Nobody really cares for the exact rules and a simple \d\d\-\w+\.conf pattern for the filenames is more than enough in 99% of cases.

[0] https://www.zabbix.com/documentation/6.0/en/manual/appendix/...

Oh yes! Good thing if it only happens in the build server and not on deployments, but that can be "fun" enough. Bonus points if you have a culture of repeat builds being "special".

I've always seen this solved by sorting (lexicographical order of bytes, originally). First two characters were digits, for all files within a wildcard directory.

> Always check your classpaths for duplicates kids.

I recently discovered extra-enforcer-rules which can help with that, as long as your classpath comes from Maven dependencies.

Because "first" and "last" don't convey priority. You have to know how the configuration parser works with defaults and overrides. Does the parser stop reading after the first match? Do later configuration directives override previous ones or are they appended to them, and if they are appended, in which order are they evaluated?

First and last are only ambiguous if we make the order ambiguous though. There's no reason for the order to be ambiguous, and if it is then obviously we can't use first/last. I don't think that first/last are either ambiguous or implementation dependent concepts. The other questions are simply other questions, and aren't directly related to whether the ordering is ambiguous or not.

There are plenty of applications that read a set list of configuration files and use either the first or last setting found. I agree with the responder that the last setting usually makes more sense.

> and if they are appended, in which order are they evaluated?

This would be a problem for first wins too.

Also: processes / running programs should provide a straightforward way to show their configuration.

While "sshd -T" shows the effective configuration from all configuration file parsing, there is no straightforward way to see the configuration of the/a currently running sshd process.

This has footgun potential, as someone could think their sshd is secured and hardened by checking with "sshd -T", while the actually running sshd process still uses old configurations.

Apache also makes it needlessly hard to find out its effective (and active / currently running) configuration. I don't know of any better way then awkwardly using mod_info ( https://httpd.apache.org/docs/2.4/mod/mod_info.html ) to get such insights.

Not sure I get the reasoning here. That requires you to actually go and proofread the configuration of each and every running process, at which point it's easier to just verify the effective configuration on disk and restart the service.

SSH is a special case because it leaves active sessions alive when it restarts, but that doesn't generalize to other daemons. Just restart if in doubt.

This is good advice and my own software is following it:

$ oq info cfg

Looking at the following paths (the last wins)

/opt/openquake/oq-engine/openquake/engine/openquake.cfg

/opt/openquake/venv/openquake.cfg

However, I do not think there is any standard API to follow, I used "info cfg" but many alternatives were possible. So sysadmins have to explore to find out which is the introspection command to give. Not ideal, but I don't think there is a solution.

I would look for something like this in the help output, i.e. --help or -h. And of course the manpage if you do supply one.

Another good place for that is the --version output, if your --version is one of those verbose multi-line ones (instead of a basic single-line output).

If you pass -verbose switch to any of my programs you'll definitely see a line similar to this:

    verbose: Configuration file found at $PATH.

There are a couple of places I look for my config files: /etc, ~/.config and "./conf", at that order. The last one accelerates testing a great deal.

And every configuration that overrides the built-in defaults are reported one by one.

Also, I started to detail how configuration file works in README.md file, under its section now.

Lastly, I decided to fix all of my tools to work on same specs from now on, regardless of the programming language they're written in:

1. Use TOML, and TOML only.

2. Config file overrides the defaults, Command line flags overrides the configuration file.

Config file overrides the defaults, Command line flags overrides the configuration file.

Don't forget "user configuration file overrides system configuration file (but is overridden by command line)".

In tarsnap the priority (lowest wins) is:

0: Command line

1: ~/.tarsnaprc

2: /etc/tarsnap.conf (may be in a different system etc depending on platform standards)

3: tarsnap defaults.

Where would configuration options via environment variables fit in? I can see between 0 and 1, or 1 and 2, but can't quite decide which is better.

Not relevant to tarsnap (and in general something I would be very careful about adding since people often don't realize what's in their environment) but I would say that command line options take priority over the environment and the environment takes priority over all configuration files.

+1 on both points - support for env and handling env priority

Nice example, which opened a small rabbit hole for me: what does "first occurrence wins" mean for options that are meant to be used multiple times, such as "Port" in sshd_config. The man page only says "Multiple options of this type are permitted.", but now I wonder what happens when I put my own Port into /etc/ssh/sshd_config.d/whatever.conf but leave "Port 22" in sshd_config? Or is the (new since bookworm) default sshd_config commenting it out for this reason?

More generally, for multi-options do we maybe want "first file wins"?

Reading in general to more specific configuration directories in order and using the last value read makes the most sense to me. This is usually how I write my own applications.

Anyway, I thoroughly document the order and precedence, but the suggestion from Chris' blog to report more detail about the process is a good one.

>>>Some programs use a "last occurrence wins" approach, while other programs (e.g. sshd) use a "first occurrence wins" approach.

A bunch of applications at my job use a config-loading library that will fail on startup if the same option is set to two different value (on different lines). It's certainly frustrating at times...but it's also prevented a huge amount of ambiguity.

That would actually be my preference TBH. Failing quickly at startup is usually far better than unpredictable behaviour later on, particularly if it's a long-running service. Though I've always thought that the behaviour of tools when launched interactively by the user, vs launched via an automated background process should probably be different - it may well be that if a user/administrator isn't aware it failed on machine startup the consequences of it not being running at all are worse than an ambiguous configuration setting.

I am incredibly torn on if this is brilliant or would have me cursing the developer's name.

I prefer the "setting it twice to different values is an error" mode, which isn't especially common.

Also, they should report where logs are written.

logs always go to stdout/stderr

hopefully!

When discussing, let's keep in mind what really mature configuration can look like:

https://docs.spring.io/spring-boot/docs/2.1.13.RELEASE/refer...

This is a big ridiculous, but the fact is that the entire order of processing there is a result of practical need evolved over likely thousands of organizations.

There are unit testing concerns, dev/test/stage/prod concerns, CLI args, OS vars, config files, IDE vs non-ide concerns, and more. And even some references to databases (if you squint and see JNDI as a database).

So yeah, finding out where property value X was set can be a bit maddening oftentimes.

"Configuration" is an afterthought in most languages and frameworks in initial iterations. So bespoke / one-off solutions will explode early on. As the original complaint details, "configuration" isn't just a simple property file. It gets complex pretty quickly, and a good configuration subsystem would be nice:

- supports most of the concerns above

- provides a means for logging the final config value set and WHERE IT SOURCED FROM

But... oh wait, config values aren't just key-value. Wait until you get to trees of values or other more complicated config. Then value overlaying gets pretty hard, and resolution code gets complicated too.

Think of infrastructure as code frameworks. The configuration is insanely deep and complicated. Many complicated systems have "configuration languages", and they are probably Turing Complete.

And once your configuration system is complicated enough ... your configuration system kind of implicitly needs ITS OWN configuration system.

So maybe the first config layer is actually a bootstrap for loading all the configuration info.

And.... then you have config values you need to secure. Hooo boy, now you have all the complexity of high security systems interfaces: authentication and authorization, decryption and hashing.

And invariably there are binary blobs encoded in text friendliness.

It gets crazy. Fast.

BUT THE AMOUNT OF SIMILARITY IS NUTS. NUTS. This is begging for a common approach or standard or library.

It's crazy that a year from now, no one is ever going to be thinking about this garbage ever again. By then, enough programmers will have discovered how insanely good GPT4 is at doing software configuration, and in the absence of the right answer, reading these verbose docs to finally just tell you what to do.

lol

There should be an option to report where any currently active non-default configuration settings actually came from, individually. (including if that's shell environment vars, etc).

I forget what tool I use regularly that actually does this. Is it git?

It would also help if an active process said if it was running using the current config.

For example, if a change has been introduced to a config file, but the process has not been restarted to incorporate those changes, this should be conveniently mentioned.

Reply on news.ycombinator.com