Everything that uses configuration files should report where they're located
utcc.utoronto.ca
utcc.utoronto.ca
Some programs use a "last occurrence wins" approach, while other programs (e.g. sshd) use a "first occurrence wins" approach.
Buried in https://man.freebsd.org/cgi/man.cgi?sshd_config(5) and awkwardly expressed:
"Unless noted otherwise, for each keyword, the first obtained value will be used."
That's why e.g. Debian's /etc/ssh/sshd_config ( https://salsa.debian.org/ssh-team/openssh/-/blob/debian/1%25... ) starts with
Include /etc/ssh/sshd_config.d/*.conf
While "sshd -T" shows the effective configuration from all configuration file parsing, there is no straightforward way to see the configuration of the/a currently running sshd process.
This has footgun potential, as someone could think their sshd is secured and hardened by checking with "sshd -T", while the actually running sshd process still uses old configurations.
Apache also makes it needlessly hard to find out its effective (and active / currently running) configuration. I don't know of any better way then awkwardly using mod_info ( https://httpd.apache.org/docs/2.4/mod/mod_info.html ) to get such insights.
SSH is a special case because it leaves active sessions alive when it restarts, but that doesn't generalize to other daemons. Just restart if in doubt.
https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity...
Some examples:
1. #mydiv overrides .mydiv - IDs are more specific than classes
2. .outerdiv .innerdiv overrides .innerdiv - nested selectors are more specific
3. .mydiv overrides div - classes are more specific than elements
And a lot of other rules.
I can’t imagine I’d want this level of complexity in config files.
Rather specificity is a triplet of (ids sub-selectors, class sub-selectors, element sub-selectors). When attributes conflict, the one whose rule has the more specific selector wins.
The specificities of the rules you wrote out are (1,0,0), (0,1,0), (0,2,0), (0,1,0), (0,1,0), and (0,0,1).
In-line attributes (@style) beat out out-out-of line ones except if they’re !important. Specificity also disambiguates between !important attributes.
Lots of people don’t want it in their CSS either. It’s generally considered good practice to make everything the same specificity and avoid !important.
Sure, haven't seen that in a while in well structured code bases.
> make everything the same specificity
How would you achieve same specificity on everything? Only use either ids or classes? No type selectors / pseudo-classes / pseudo-elements / multiple selectors? I can't imagine how that would be feasible and am not aware of anyone recommending this.
:is(selector_1, :where(selector_2))
Has always the specificity of selector_2, and if you choose an impossible selector for selector_1 (the easiest approach is to give multiple ids or use non existent classes/elements) it only select by selector_2
Specificity is at the heart of CSS, you're not supposed to design rules with equal specificity
IIRC, there's an important (heh) exception to that: user stylesheets. Without the "!important" override on the user stylesheet, the site stylesheet wins; with the "!important" override on the user stylesheet, the user stylesheet wins, even if the site stylesheet also had an "!important" for that rule.
Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.
E.g. from systemd.unit manpage ( https://www.freedesktop.org/software/systemd/man/systemd.uni... ):
>...
> In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc/ take precedence over those in /run/ which in turn take precedence over those in /usr/lib/. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.
> ...
And this is just an excerpt which doesn't cover the details. Read the manpage for all the hilarity.
Ansible does apply variable precedence, and you might have a use for it. Here is the order of precedence from least to greatest (the last listed variables override all other variables):
1 command line values (for example, -u my_user, these are not variables)
2 role defaults (defined in role/defaults/main.yml) 1
3 inventory file or script group vars 2
4 inventory group_vars/all 3
5 playbook group_vars/all 3
6 inventory group_vars/* 3
7 playbook group_vars/* 3
8 inventory file or script host vars 2
9 inventory host_vars/* 3
10 playbook host_vars/* 3
11 host facts / cached set_facts 4
12 play vars
13 play vars_prompt
14 play vars_files
15 role vars (defined in role/vars/main.yml)
16 block vars (only for tasks in block)
17 task vars (only for the task)
18 include_vars
19 set_facts / registered vars
20 role (and include_role) params
21 include params
22 extra vars (for example, -e "user=my_user")(always win precedence)that is a RedHat-isim
I get that hating RH is in vogue currently, but let's keep it grounded in reality please.
EDIT: And for the record, these Ansible precedence rules are not a bad thing. Once you are somewhat familiar with them, it allows you to write incredibly flexible and extensible playbooks and roles.
I am also ever more deeply concerned about IBM’s hell-bent purge of all the good left at Redhat. I had thought it would take them less time to destroy Redhat, but I’m sure they wanted to make sure they spent time “listening” before making all these ecosystem-eroding changes.
".d" directories are a well-established pattern to split a complicated config file into logical units. In the case of something like systemd this is absolutely essential, as you wouldn't want the config for your web server in the same file as the mail server.
/usr/lib is the location where the package manager will drop config files. This is the default, distro-provided configuration. /etc/ is where the administrator can place config files, providing system-specific overrides. /run/ is for files "created at runtime", I am not quite sure what the point of those is.
Systemd is also a useful demonstration of how to show people where configuration comes from -- `systemctl status <unit>` will include a list of all the files found during lookup, and `systemctl show <unit>` will give you the resolved unit configuration including all the defaults.
>Read the manpage for all the hilarity.
If you find it hilarious it's because you don't understand the reason. /usr belongs to the distribution and /etc belongs to the user. The distribution doesn't touch /etc and the user doesn't touch /usr.
Upstream ships a unit with its own opinion of the defaults. That unit goes in /usr.
Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file.
User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.
Some units are created at runtime on demand and thus ephemeral, like mount units generated by generators. These go in /run. These are functionally "system-provided units" because users must be able to override them, so they're below /etc in the hierarchy.
The "defaults" and "overrides" we're talking about are both at the level of the individual setting as well as at the level of the whole file. If a user wants one of the settings to be different, they can add a new 99-foo.conf file that unsets / changes it. If they want a whole file to be different, they can add a new file with the same name that is empty / has different values for its settings.
BTW `systemctl cat whatever.unit` will print all the files that systemd considered in the order that it considered them.
>And this is just an excerpt which doesn't cover the details.
The only detail your excerpt doesn't include is the consideration of `service.d/`, `socket.d/` etc directories which provide overrides for all units of that type. It's just one point, not commonly used, and not that big of a deal either.
So it's not hilarious statements like this make it hard to understand the reason? I almost cannot tell whether your whole post is sarcasm, or just showcasing irony.
If you're deploying an especially big application, you may also prefer /opt/$APPNAME/{bin,etc,usr,...} (A full filesystem tree, basically).
We manage a fleet of systems at work, and I don't remember editing or touching a file under /usr, unless the period I was developing .deb or .rpm packages to deploy on said fleet.
There are many written and unwritten rules in Linux world, and sometimes people disregard or ignore these rules so blatantly, so I can't understand whether the post I'm replying to is a sarcasm or irony.
https://access.redhat.com/documentation/en-us/red_hat_enterp...
There's also a section for /usr.
• Filesystem Hierarchy Standard <https://www.pathname.com/fhs/>
• For Debian (and derivatives): <https://www.debian.org/doc/debian-policy/ch-opersys.html#fil...>
<https://manpages.debian.org/stable/manpages/hier.7.en.html>
If so, I don't think anybody will disagree.
But if you meant that's a large problem, then well, no it isn't. There are 7 of those badly named ones. It's a matter of reading half a page of text to see what they do, and after 2 or 3 times you read it as reference, you have them memorized.
If would be nice to solve the issue, but it isn't anybody's priority.
But yeah, ease of use is nobody's priority
In many ways I love it, of course. It's what I grew up on, and I'm sure I'll have a small Linux cluster in the retirement home, because that's how computers are "supposed to be". But I think the old paradigm has only lasted so long because we have so much computing power to burn.
Well, most of us. A good check on this is a podcast where Dr Kevin Bowers is interviewed: https://www.youtube.com/watch?v=GlzWM5GKQT8
He works for a financial trading firm, where the competitive nature of it means that every processor cycle counts. For him, older paradigms are perplexing and irritating, because they are major barriers to getting the most out of the hardware.
In particular, in our own version of the system, there is a directory "/usr" which contains all user's directories, and which is stored on a relatively large, but slow moving head disk, while the othe files are on the fast but small fixed-head disk. — https://www.bell-labs.com/usr/dmr/www/notes.html
> User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc.
Larry Wall elegantly solved these problems 38 years ago with the "patch" utility. Debian is a good example of how its use has been standardized to solve these problems.
The patch approach to managing distribution-provided configuration files is superior to the systemd unit approach for at least three reasons:
- Non-experts can read the configuration files from top to bottom in linear order, so the documentation can be inlined by the distribution provider, which means they can add maintainers notes as necessary.
- It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.
- The implementation of all this is factored out and standardized across 1000's of packages, so it only needs to be implemented once and understood once.
> - It also works better than the systemd unit style of configuration for sophisticated users. In particular, if you add a now-incompatible option to the file in the suggested place (next to the documentation and commented out default version of the setting), then the package management system can detect that you've made a potentially breaking change before installing the new software, and ask you what to do. This works in the absence of sophisticated data modeling, schema mapping tools, etc.
Anecdata, but still: Every single time the package manager asked me what to do, it was not because of a breaking change, but simply because the base file changed in a way that the patch didn't apply cleanly anymore. That is, I, the user, had to work around limitations of the patch format. If the distro configuration and the user configuration had been separate files, things would have just worked fine.
I think it's hilarious that you think users are supposed to understand "the reason" at this level of detail.
When we make a thing, we make choices about who it's for. Sometimes those choices are explicit. But this is a good example of an implicit choice. Clearly it grew into this shape for reasons, but those reasons are about the people doing the making, not the people using it.
Even if we grant that all of this hierarchy is necessary, it still imposes a large burden on the person trying to figure out "why is X doing Y?" And that might be fine if the makers said, "Well this is going to be hard, so what can we do to make it easy?"
But denying that users will reasonably find this hilarious (or frustrating or impossible) is just digging the hole deeper. Telling people that their understandable reactions are wrong doesn't change their reactions, it just gets them to stop mentioning the problems around you.
Systemd has tools for that, such as systemctl cat which shows all applicable configuration and what files they come from.
Personally I have found the systemd hierarchy to be very useful. It allows me to create overrides for certain properties in a Unit without having to worry about keeping the rest in sync with upstream, or my package manager overwriting my configuration. And it allows me to have different independent packages (or chef recipes or ansible playbooks) that create different override files for the same unit without stepping on each others toes.
Yes it is a little complicated, but for most end users, you don't really need to worry about most of the complexity, you just need to know where you put your overrides. And for distributors and power users, that complexity serves a useful purpose.
Sure, it could be that systemd does have amazing tooling for helping users with the burdens they have created. Or, given the very mixed feelings about systemd, maybe they've tried to be user focused but not hit the mark. I don't know enough to say.
But my point is that either way responses of the form "it's so easy, you just have to remember [9 paragraphs of detail users don't care about]" are part of the problem, not the solution.
But are all those requirements actually necessary for the job it's doing? Or have its developers imposed artificial constraints through their choices in designing systemd that make it more complex than it could otherwise be?
A distribution that doesn't own any directories like this effectively can't make any changes. It's like writing a class without private fields.
I am not an expert but I think in general systemd has a lot of complexity but it's to handle existing issues in a better way. Some of the older init systems might be simpler to describe or get started but lead to more confusing situations in the long run.
I find the best way to think about systemd is as the result of some people that probably knew the most about init systems remaking init to be consistent, documented, and handle a bunch of new features that different newer init systems had used to good effect.
Then, because regardless of how much those people knew the vast mofority of information about needs was sequestered in 1000 different teams and projects, they had to add a bunch of new features. Repeat that 100 times, and you get what we have today, which is a system that mostly works (and works well if you actually adopt their preferred patterns), but one that also has the craziest and hardest to remember directive names I have ever seen, where slight nuance about how they function means entirely different params.
In a perfect world it would be rewritten with the same features but with all the knowledge of what features are actually needed known and consistently implemented with sane naming from the beginning, but nobody wants to go through that again.
Assuming that's how the history went, fine. Everything has a history, and we have to deal with the now.
But my point is that we are not going to deal well with the now, let alone the future, unless we recognize the impact the history has on people. So if people defensively shout "that's not funny" and imply users are wrong for being bothered by the mess, then we're on track to build another mess. We can't do better until we are honest about why it didn't turn out as well as we would have liked.
I think it's less that than that there were real advances they were after, which they delivered (Poetterings original essays on what it can and should do and where the ideas came from, such as OS X's LaunchD and some things from Windows, etc) referred to numerous things that every sysadmin I knew at the time that read it was excited if could be delivered (but none of use really appreciated what it meant to deliver them).
The issue is not that they initially built it to their own desired, but that the way SysV init worked with just launching bash scripts to start and control the process that eventually should be running meant that a lot of crazy behavior was put into scripts along the years, because when you can write a program to control your program, sometimes it's easier to just put special behavior into your launch script than to patch the real code in question in some way.
When this crashed against systemd's desire to know what's actually running and control the environment perfectly (which is beneficial, it was always a pain to have a process that started fine from your root user shell but not on startup), and strongly disincentivize programs between systemd and the process in question (which a bash script is) for reasons of process and resource tracking (Linux kernel cgroups were just starting to widely be used beneficially around this time), there started to be a mass influx of feature requests for behavior that used to be handled by dedicated bash scripts previously, but now systemd needed to account for.
Combine that with systemd adding real dependency tracking (as well as start order tracking, which is related but not quite the same, leading to the difference in Wants and Requires), and what we have is a system that's just trying to do something vastly more complicated than what SysV init did previously (and that's before we get to the scope creep which caused them to want to take over some other early boot processes to they could be deterministic or whatever, which is also a PITA for many people).
If you're actually interested in the history if this, I recommend reading "systemd, 10 years later: a historical and technical retrospective" at https://blog.darknedgy.net/technology/2020/05/02/0/. That's where most my info on the details come from. I was around in the beginning and was excited to see systemd start, but I didn't actually use it until far later since most the systems I administer don't get changes like that immediately (and it was in an extremely slow release cycle at that time).
If you read the above you may come away with a different perspective than me, but I will say that systemd is very consistent in its operation and very robust in how it tracks processes and whether there's been a problem and restarts are needed, etc, as long as you actually adopt their preferences (which is to say run everything foreground and let systemd deal with STDIN/STDOUT and environment, and have a good pid to start with). If you do that, writing a new service file is extremely easy. Far more easy than it ever was with SysV init. It's a different paradigm, and sometimes it can be a pain when some process doesn't want to play nice, but mostly it just works once you stop fighting it, which is nice, and cuts down on admin time figuring out problems (especially since it just grabs all STDOUT and STDERR for you so random error messages that don't go to logs aren't lost either).
I should say that "real advances they were after" is not at all incompatible with "made it mainly to please themselves". The things that pleased them may well have been "real advances"; they often are. The question is the extent to which they were prioritizing the needs, capabilities, and interests of the whole audience being served, not just the things they personally thought important or preferable.
Let's say you need to pass a special flag to sshd on your system, a bad approach would be to go edit the distro shipped sshd.service file. Your change will work until the distro ships an updated file and you forget to hack in your change again.
Instead place a sshd.service drop-in config file in the appropriate place to customize the service as necessary. Your config tweak takes precedence over the distro shipped config, but doesn't live inside it or conflict with upstream changes.
$ oq info cfg
Looking at the following paths (the last wins)
/opt/openquake/oq-engine/openquake/engine/openquake.cfg
/opt/openquake/venv/openquake.cfg
However, I do not think there is any standard API to follow, I used "info cfg" but many alternatives were possible. So sysadmins have to explore to find out which is the introspection command to give. Not ideal, but I don't think there is a solution.
https://docs.spring.io/spring-boot/docs/current/reference/ht...
I have been leaning towards a different approach. Rather than reading multiple config sources at runtime, read them at build time, and emit a single config file with everything fully resolved. Put that on the filesystem, and have the app read it. You have an easy canonical reference for the configuration, with no need to expose anything. The lifecycle of the config doesn't have to match that off the app binary, so it could (and probably should) have a separate build and release process.
The problems with this approach when it's enabled is that log aggregators (e.g splunk) can end up including stuff like passwords which shouldn't be visible to the group able to view those logs, so some care is needed.
Logging has its merits, like discoverability when you stare at it without suspecting configuration to be the culprit, but I believe that there can and should be more.
hopefully!
I recently discovered extra-enforcer-rules which can help with that, as long as your classpath comes from Maven dependencies.
If you don't care (eg [0]) then do nothing and let the user bang it's head and reinvent the wheel.
If you care - just sort it and mark it as 'platform dependent' or somehow. Nobody really cares for the exact rules and a simple \d\d\-\w+\.conf pattern for the filenames is more than enough in 99% of cases.
[0] https://www.zabbix.com/documentation/6.0/en/manual/appendix/...
This would be a problem for first wins too.
There are plenty of applications that read a set list of configuration files and use either the first or last setting found. I agree with the responder that the last setting usually makes more sense.
Rejecting multiple definitions is better, unless you care about DoS (I would be pissed if I got locked out of ssh access due to an option being defined twice, for instance).
At least the inspector in Safari (and maybe other WebKit browsers) does something similar for CSS.
I’m all for it. It’s a pain when writing config systems (no longer just key/value, but key+value+meta), but very helpful. It can be a pain for things like JSON where libraries don’t give you that type of diagnostic information easily, however.
verbose: Configuration file found at $PATH.
And every configuration that overrides the built-in defaults are reported one by one.
Also, I started to detail how configuration file works in README.md file, under its section now.
Lastly, I decided to fix all of my tools to work on same specs from now on, regardless of the programming language they're written in:
1. Use TOML, and TOML only.
2. Config file overrides the defaults, Command line flags overrides the configuration file.
Don't forget "user configuration file overrides system configuration file (but is overridden by command line)".
In tarsnap the priority (lowest wins) is:
0: Command line
1: ~/.tarsnaprc
2: /etc/tarsnap.conf (may be in a different system etc depending on platform standards)
3: tarsnap defaults.
More generally, for multi-options do we maybe want "first file wins"?
Anyway, I thoroughly document the order and precedence, but the suggestion from Chris' blog to report more detail about the process is a good one.
And the thing with developers is that they make lousy scientists. If they want something to work, they’ll take any information that supports that theory and run straight to the repository with it.
The trick is not to fool yourself, and you’re the easiest person to fool.
For example, if a change has been introduced to a config file, but the process has not been restarted to incorporate those changes, this should be conveniently mentioned.
https://docs.spring.io/spring-boot/docs/2.1.13.RELEASE/refer...
This is a big ridiculous, but the fact is that the entire order of processing there is a result of practical need evolved over likely thousands of organizations.
There are unit testing concerns, dev/test/stage/prod concerns, CLI args, OS vars, config files, IDE vs non-ide concerns, and more. And even some references to databases (if you squint and see JNDI as a database).
So yeah, finding out where property value X was set can be a bit maddening oftentimes.
"Configuration" is an afterthought in most languages and frameworks in initial iterations. So bespoke / one-off solutions will explode early on. As the original complaint details, "configuration" isn't just a simple property file. It gets complex pretty quickly, and a good configuration subsystem would be nice:
- supports most of the concerns above
- provides a means for logging the final config value set and WHERE IT SOURCED FROM
But... oh wait, config values aren't just key-value. Wait until you get to trees of values or other more complicated config. Then value overlaying gets pretty hard, and resolution code gets complicated too.
Think of infrastructure as code frameworks. The configuration is insanely deep and complicated. Many complicated systems have "configuration languages", and they are probably Turing Complete.
And once your configuration system is complicated enough ... your configuration system kind of implicitly needs ITS OWN configuration system.
So maybe the first config layer is actually a bootstrap for loading all the configuration info.
And.... then you have config values you need to secure. Hooo boy, now you have all the complexity of high security systems interfaces: authentication and authorization, decryption and hashing.
And invariably there are binary blobs encoded in text friendliness.
It gets crazy. Fast.
BUT THE AMOUNT OF SIMILARITY IS NUTS. NUTS. This is begging for a common approach or standard or library.
A bunch of applications at my job use a config-loading library that will fail on startup if the same option is set to two different value (on different lines). It's certainly frustrating at times...but it's also prevented a huge amount of ambiguity.
I forget what tool I use regularly that actually does this. Is it git?
You can think of everything as configuration. Parameters of a function are just configuration for that function.
What we need is traceability of every value in a system, not just configuration. You click on any value and you see a graph/chain of all the transformations this data goes through right back to the source of truth. This is how we should code. Think about how difficult this is to do today with your current IDE/editor.
Every task in software is wanting to change what is displayed in a user interface of some sort. Usually you have to make your way through a labyrinth to figure out what to change where, and also what side-effects will occur.
If this is not smack bang in front of your face when you are coding, then you are just adding more and more tangled data flows and complexity to your system.
lately i’ve been playing with prolog and revisiting computer science/computer programming before imperative languages drew a curtain on innovation and i think we have lost a lot. our modern programming isn’t the programming the founders had in mind, it seems.
But imperative ones create a bit of a problem. Nothing impossible to deal with, but it becomes harder.
But any choice of language, the OS can enforce it too, and maybe in a much easier way than messing with your language.
Designers will put much more thought into how best to interact with a framework, so start with a small, opinionated interface then consider extension feature requests on a case by case basis
I think some functional programming languages solve that by running a good chunk (if not all) of the application in a single trace.
And, of course, async/non-blocking calls, as tracing a call along different threads or promises may not be available all the time.
Chastised, Anton took his leave from his master and returned to his cell, intent on studying closures. He carefully read the entire "Lambda: The Ultimate..." series of papers and its cousins, and implemented a small Scheme interpreter with a closure-based object system. He learned much, and looked forward to informing his master of his progress.
On his next walk with Qc Na, Anton attempted to impress his master by saying "Master, I have diligently studied the matter, and now understand that objects are truly a poor man's closures." Qc Na responded by hitting Anton with his stick, saying "When will you learn? Closures are a poor man's object." At that moment, Anton became enlightened."""
For example in php class A { public $a = new B;} is not valid
Such a restriction makes no sense in a "closures are a poor man's object" world.
I think there is still hope on that front. Following structured concurrency patterns like the supervisor model should handle it. See the discussion about "Notes on structured concurrency, or: Go statement considered harmful" [1].
it's certainly not a given
No different from an object that got constructed and then a method called on it.
i thought you meant state that is created from a call stack and persists after that call stack returns -- which is something else
It's mostly feasible in other languages to, but probably not with the same level of rigor.
Anybody here from JetBrains?
Am I the only one that thinks pull requests (asynchronous) are a more appropriate medium of collaboration for this kind of work that requires strong consistency between elements?
This kind of reminds me of the Whyline: https://www.cs.cmu.edu/~NatProg/whyline.html
Yippee. /s
Isn't that just a debugger?
> spack -e default config blame packages
--- packages:
spack/environments/default/spack.yaml:59 fftw:
spack/environments/default/spack.yaml:60 require: ~mpi
spack/etc/spack/defaults/darwin/packages.yaml:17 all:
spack/defaults/packages.yaml:18 compiler: [apple-clang, gcc, clang]
spack/defaults/darwin/packages.yaml:20 providers:
spack/defaults/darwin/packages.yaml:21 unwind: [apple-libunwind]
spack/defaults/packages.yaml:20. blas: [openblas, amdblis]
spack/defaults/packages.yaml:21 jpeg: [libjpeg-turbo, libjpeg]
spack/defaults/packages.yaml:22 lapack: [openblas, amdlibflame]
spack/defaults/packages.yaml:23. mpi: [openmpi, mpich]
spack/defaults/darwin/packages.yaml:22 apple-libunwind:
spack/defaults/darwin/packages.yaml:23 buildable: False
spack/defaults/darwin/packages.yaml:24 externals:
spack/defaults/darwin/packages.yaml:27 - spec: apple-libunwind@35.3
spack/defaults/darwin/packages.yaml:28 prefix: /usr
For the curious, there is more on scopes and merging [2]. This isn't exactly easy to do -- we use ruamel.yaml [3], but had to add our own finer-grained source attribution to get this level of detail. There are also tricky things to deal with to maintain provenance on plain old Python dict/str/list/etc. objects [4,5].
[1] https://github/spack/spack
[2] https://spack.readthedocs.io/en/latest/configuration.html
[3] https://yaml.readthedocs.io/en/latest/
[4] https://github.com/spack/spack/blob/95ca9de/lib/spack/spack/...
[5] https://github.com/spack/spack/blob/95ca9de/lib/spack/spack/...
However, there is still a lot of room for improvement in compilers and debuggers. If I could tell the compiler to instrument my code to track the values that I care about, and then have the debugger visualize the flow of information through the system, that would be very useful. It wouldn't guarantee that the data flow analysis applies to subsequent runs of the program, but it could reduce the amount of time spent reading and debugging code.
I think there is a new era yet to come.
> probably too complex to visualize in a usable way
You can always abstract the complexity. A simplification of the control flow graph at the function level.
As an example, I want to understand the Postgres codebase better...there are a lot of very visualizable things going on there. If you look at lecture slides on relational DBMS, there are lots of diagrams...so why can't be view those animating diagrams as the code executes.
I was thinking of instrumenting the code to get a trace of every function call and its arguments, and then filter that and visualize it.
The key to this approach working though is a think ultimately we need to write our code and organize our code such that it can be easily visualizable. The purpose of code is for humans to understand it, otherwise we are writing assembly.
Static-checking tools and strongly-typed languages: "Are we a joke to you?"
Finding and removing bugs before the code runs is still debugging.
Man, you would love functional programming. I'm' not even joking.
Microsoft IntelliTest (formerly Pex) [1] is internally using Z3 constraint solver that traces program data and control flow graph well enough to be able to generate desired values to reach given statement of code. It can even run the program to figure out runtime values. Hence the technique is called Dynamic Symbolic Execution [3]. We have technology for this, just not yet applied correctly.
I would also like to be able to point at any function in my IDE and ask it:
- "Can you show me the usual runtime input/output pairs for this function?"
- "Can you show me the preconditions and postconditions this function obeys?"
There is plenty of research prototypes doing it (Whyline [4], Daikon [5], ...) but sadly, not a tool usable for the daily grind.
[1] https://learn.microsoft.com/en-us/visualstudio/test/intellit...
[2] https://link.springer.com/chapter/10.1007/978-3-642-38916-0_...
[3] https://queue.acm.org/detail.cfm?id=2094081
In my experience most actuaries I interact with maintain truly horrendous spreadsheets with labyrinthine complexity.
The distributed aspect is a bit more difficult though. I don't know if there is a system that truly got this right. Maybe Erlang's BEAM VM together with a good debugger I don't yet know about.
Sweeping generalizations rarely age well.
> You can think of everything as configuration. Parameters of a function are just configuration for that function.
No, parameters of a function are contextual collaborators which participate in a function implementation. Configuration canonically influences a program aside from its individual run-time interactions.
Is a value given to the cosine function a configuration? Words matter.
> Every task in software is wanting to change what is displayed in a user interface of some sort.
This is a myopic perspective. Software processes data into information which provides value in context. This may never be "displayed in a user interface."
To wit, the network gear involved in my posting this reply most certainly qualifies as a "task in software." So, too, is the software running in the power companies involved. It would be quite a stretch to categorize them as "wanting to change what is displayed in a user interface."
$ redis-cli INFO
...
executable:/Users/antirez/hack/redis/src/./redis-server
config_file:/etc/redis/redis.conf
...
There are commands to get reports about latency and memory usage, performing a self-analysis of the instance condition. You can see counters for commands called, and even obtian a live stream of the commands the instance is executing.
This should all be obvious features to make system software approachable, observable, and to make people able to act easily when shit happens, without having to guess where the configuration file is from the system startup scripts or alike. UX is not only about GUIs.
I had to do this to figure out where Cold Steel was reading save files from years back: https://vaughanhilts.me/blog/2018/02/16/playing-trails-of-co...
https://8thlight.com/insights/dtrace-even-better-than-strace...
Feel free to reference my feedback entry FB12061147 if you're reporting the same.
It’ll show all file events. No need to disable SIP. SIP is doing a lot of good work for users and unless you’re doing kernel work or low level coding I’d keep it enabled. Obv. There are other cases but for the general public keep it on.
https://mohit.io/blog/fs_usage-trace-file-system-calls-on-ma...
"The best way to hurt somebody who's lost everything is to give him back something that is broken."
For us, this thing is MacOS. I miss dtrace every damn day.
DTrace allowed you to ask the damn os what it was doing, since the man pages are random and do not match the command line help, new daemons constantly appear with docs like this:
NAME rapportd – Rapport Daemon.
SYNOPSIS Daemon that enables Phone Call Handoff and other communication features between Apple devices.
Use '/usr/libexec/rapportd -V' to get the version.
So while this can be a useful hack, it doesn't always work.
Though DTrace is "only" available on Windows, Mac OS X, Solaris, illumos, and FreeBSD.
Oracle has relicensed DTrace to be Linux-friendly and even made kernel patches, but it'll probably never end up in the mainline kernel.
Is it stable and production ready for heavy duty workloads?
[1]It used to have its own kernel extension but is eEBF based these days.
┌────────────────────────┤ Configuring Secure Boot ├────────────────────────┐
│ │
│ Your system has UEFI Secure Boot enabled. │
│ │
│ UEFI Secure Boot requires additional configuration to work with │
│ third-party drivers. │
│ │
│ The system will assist you in configuring UEFI Secure Boot. To permit │
│ the use of third-party drivers, a new Machine-Owner Key (MOK) has been │
│ generated. This key now needs to be enrolled in your system's firmware. │
│ │
│ To ensure that this change is being made by you as an authorized user, │
│ and not by an attacker, you must choose a password now and then confirm │
│ the change after reboot using the same password, in both the "Enroll │
│ MOK" and "Change Secure Boot state" menus that will be presented to you │
│ when this system reboots. │
│ │
│ If you proceed but do not confirm the password upon reboot, Ubuntu will │
│ still be able to boot on your system but any hardware that requires │
│ third-party drivers to work correctly may not be usable. │
│ │
│ <Ok> │
│ │
└───────────────────────────────────────────────────────────────────────────┘
Hitting C-c (Control-c) has no effect.
There are validity rules for the password, but they are presented only after you've entered an invalid password, twice, the second time to confirm.
After this installation proceeds, then terminates with a warning:
DKMS: install completed.
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
W: Operation was interrupted before it could finish
Distro was Ubuntu 20.04.
These days Proton makes a lot of this unnecessary. I hope Trails games will continue to at least function on Linux. :)
https://www.brendangregg.com/blog/2014-07-25/opensnoop-for-l...
https://github.com/brendangregg/perf-tools/blob/master/opens...
ps wxa | grep <cmd> # or pgrep
strace -f -f -e trace=file <cmd> | grep -v '-1 ENOENT (No such file or directory)$'
ps wxa | grep <cmd> # or pgrep
strace -f -f -e trace=file <cmd> 2>&1 | grep -v '-1 ENOENT (No such file or directory)$'
In lieu of strace, IDK how fast `ls -al /proc/${pid}/fd/*` can be polled and logged in search of which config file it is; almost like `lsof -n`:
pid=$$ # this $SHELL's pid
help set; # help help
(set -B; cat /proc/"${pid}"/{cmdline,environ}) \
| tr '\0' $'\n'
(while 1; do echo \#$pid; ls -al /proc/${pid}/fd/*; done) | tee "file_handle_to_file_path.${pid}.log"
# Ctrl-C
## < managed by config_mgmt_system_xyz (in path.to.module.callable) >
## </ managed> strace -f -e trace=file -o >(grep -v ENOENT) <cmd>[1] https://blog.flowblok.id.au/2013-02/shell-startup-scripts.ht...
If you don't have a way to simply tell me what your current config is, then you have no earthly right to run it.
There is a system view on top of their configuration file(s) that reports from which file and line number the currently active value was taken from.
Now, hear me out. The problem is not the location of configuration. The problem is the crippled and anemic way UNIX (and the likes) programs take configuration. These can only predictably take environment variables and command-line arguments. Both ways are crippled due to size limit (especially the command-line arguments) and structure (you cannot even have lists / arrays in environment variables). This is the reason why every application author believes they must write their own configuration, with their own (crippled) configuration format and logic for locating their configuration.
On the bright side, I think, systemd is moving towards making this mess more uniform. On the not so bright side, systemd doesn't appreciate the difficulties associated with the problem and doesn't seem to have any sort of plan moving forward. So, unit files are kind of configuration, but because the format is so crippled and anemic, they immediately added a way to source random external files for process environment etc.
Finally, systemd, even if they understood the problem well and put a good effort towards solving it are still powerless against the format in which system supplies input to processes. What really needs to change is all the family of execve, execlp and so on. This was always a stop-gap / bad idea, but it was OK for a while, before programs started to require more complex input. It should've died in the 80s... but today it is so deeply entrenched that removing it will take a very long time.
Another problem here is the approach that "everything is a file". So, the only way to provide configuration beyond command-line arguments and environment variables is a file... But, imagine an alternative reality in which OS is capable of providing basic data-structures, which can also be persisted and queried? Kind of like you do it with a relational database? I think this idea is so much not novel that it's older than most people commenting in this thread... And, even though this would make it so much easier for so many application authors to provide consistent and easy to inspect, better tested interface to their programs... nobody's doing that :) Because files are good enough.
Just take a simple task of copying a subset of some system's configuration to another one. On Unix it's just a matter of copying some files. On Windows (and our imaginary is that provides object storage) we would need to find the relevant objects, then write a script to export/import it (or mess with export/import tools for a while).
Personally I see "everything is a file" as strength not weakness. In addition to my personal preference having a centralised configuration system not only restricts in many ways, presents a potential performance bottleneck, but also is an extra point of failure.
Additionally, I think most Unix software uses a fairly simple text format for config files. There really is little reason for using anything else. If you need complex data structures use YAML. I've found many more proprietary binary config file formats on Windows (despite registry) than on Unix. So if the os would make something like this available there would still be no guarantee all apps use it. It would just be an extra thing to deal with for little benefit IMO.
Why not have files and an database like interface for them?
Even better, databases normally provide a query plan explanation, quite like what is demanded here.
But, I stick with .ini still because, 1) I have a great access module that I would have to rewrite and, 2) there's always a text editor and sometimes I'd have to add a sqlite UI. Sounds like work to me.
Please god no. JSON or TOML at most. YAML is too complicated.
Yaml only gets weird if you want to use complex objects as dict keys but that complexity is on you then.
So do JSON and TOML. If you think you need more than what they provide, something has gone very wrong.
>tools to avoid repeating yourself,
Bad. Config is not code. Settings should be explicit. DRY is not helpful.
Because of the lack of anchors/aliasing/variables etc, JSON has the nice property that you can parse it with exactly one upfront memory allocation: N bytes of JSON requires at most N words of memory.
>lord of ways to format strings so they stay readable
Bad. Too many ways: https://stackoverflow.com/a/21699210/6107981 No normal human being can remember all the rules for how a string gets parsed. Therefore it isn't actually human-readable or human-writeable, despite masquerading as such.
>arbitrary object serialization
You can write out the fields as a JSON object. If this isn't enough, then what you're trying to serialize doesn't belong in a human-readable text config file.
>Yaml only gets weird if you want to use complex objects as dict keys but that complexity is on you then.
This assumes the guy who wrote the config schema is the same as the guy who has to later use and edit it, which is seldom the case. When people do needlessly complicated config like this, they're taking an enormous stinking shit on my lawn for me to clean up.
I disagree. From experience, there are enough times that configuration requires repetition, loops, interpolation, etc, that you often end up in this gross space of either repeating chunks manually (error prone and brittle), or using code/templating to generate config (requires everyone on the codebase to use a specific toolchain).
HCL2 (used by Terraform) has been my favorite configuration system. It's just the right amount of power that I don't need copypasta or feel the need to resort to configgen. TOML is great for a single human-defined configuration for a program, but HCL does just as well at that, while also working for systems.
JSON definitely fails as human-editable. Again, you need some tooling to ensure syntax correctness.
> JSON has the nice property that you can parse it with exactly one upfront memory allocation: N bytes of JSON requires at most N words of memory.
I really don't care how much memory is allocated during parse, within reason, as long as execution is bounded, can't blow up, or otherwise misbehave.
In such a case, use a Python script to generate the configuration, and then commit that generated config into source control. Keep the script and the generated config together in the same repo. You can set up a commit hook to keep them in sync. That way you can see explicitly what the configuration is, while reducing the repetition in the editing process.
>JSON definitely fails as human-editable. Again, you need some tooling to ensure syntax correctness.
What kind of tooling are you talking about here, other than syntax highlighting? I'm really at a loss to imagine how editing JSON by hand could be hard, other than maybe the lack of trailing comma. There's just not that much to it.
I can't really understand how all of that is simpler than having your config file have a little power in itself. Especially when that power is as easy as "use ruamel." You can't enforce commit hooks so you end up making it a build failure and that's honestly really annoying to have to push again because "right I forgot to run the config generating thing." And you're one ci.skip away from messing your system up.
Like it's adding failure modes for such little gain. If you really must have the final generated output then save it as a build artifact for inspection later.
With any significant level of nesting TOML starts to obscure the structure more than it helps anything.
I tend to start with a basic K=V (“dotenv”) format. Past that though I’m going straight to YAML.
I agree that the “everything and the kitchen sink” specification is ridiculous and rarely (if ever) use much of the functionality, but as far as representing arbitrary structured data in human readable and editable text… it sucks less than the alternatives if you stay away from the weird stuff.
It's simple when the task is simple, and it becomes a lot more complex than necessary when problems become more complex.
Consider the joy of sysfs / procfs and similar. These are filesystems that, through files, expose information that is numbers, strings, lists, even trees. There's no uniformity there, no way to predict where and what sort of files with what sort of data will appear... Sometimes you get extra files with "metadata" that describes the format found in other files, sometimes it's in the same file, and sometimes it just doesn't exist. Sometimes numbers are encoded as strings, and sometimes as bytes. Booleans can be encoded as numbers, (which are either strings or bytes).
There's plenty of weird behavior that isn't described by files / their properties. For example, suppose you want to delete a RAID device, but if fails because it turns out that it's used by LVM volume, but you didn't even know you were supposed to look for that...
It gets even worse with proprietary stuff that doesn't follow conventions. Take for instance NVidia GPUs. They do appear in sysfs as PCIe devices... but most interesting properties just aren't listed there. Instead, NVidia's driver creates "devices" in udevfs (i.e. /dev) with arbitrary properties that you cannot predict because the format is whatever the vendor wanted it to be at any version of their product.
In other words, files are a very poor mechanism for describing data. In order to work with them you need conventions that aren't part of any file mechanism. Take network interface names, or block device names, stuff like eth0 or nvme0n1p1 -- you need to know the convention and how to interpret the name (i.e. the first suggests that the device is using Ethernet driver and is the first one connected, the second means that the device is using NVMe driver, it's the first device of its kind attached to PCIe bus, first namespace, first partition.) Specifically, in the case of block devices, you could discover the same information given in the file name by traversing the relevant filesystem subtree, but in some cases the information is just encoded as a file name or as it contents, but it doesn't utilize the filesystem primitives. It just relies on convention.
----
> I think most Unix software uses a fairly simple text format for config files
How did you count?
Do you mean software that is part of UNIX standard or software that runs on UNIX?
What makes the majority an interesting metric? (I.e. suppose there are 10M of different programs that run on UNIX, and 6M of them are simple enough that they don't require complicated configuration, there are still 4M of those that do, and some of them are probably used daily by millions of users.
I mean, who cares if simple programs don't need complicated configuration? The problem is the configuration of complex programs, and they are indispensible in our daily lives.
YAML is a poorly engineered format. It doesn't think about storage at all, for example (i.e. in its definition, it's just a single text blob. What if you need to split it?
The format is awful for random access: you need to read the whole thing to extract a single data item. If you need to query such a configuration multiple times you'll do a lot of unnecessary work.
YAML poorly defines numerical types (how many digits can a float have? what about integers? Are floats and integers the same thing?) YAML lists... allow mixed types. Not sure it's such a good thing from performance and correctness perspective.
YAML has unnecessary "hash-table" type. Hash-tables aren't a data format, they require function in the reader that makes asymptotic properties of hash-tables work. In fact, YAML has just a weird sub-type of list, where elements come in pairs. Since it inherited all this from JSON, there's also the ambiguity related to repeated keys in such hash-tables -- what should an application do with those nobody knows.
YAML sucks for transmission due to the ambiguity caused by repeated keys in "hash-tables". You cannot stream such a format if you have the policy that the last key wins or if you have the policy that repetition is not allowed. Also, since YAML allows references but doesn't require that the reference lexically point to a previously defined element, you may have unresolved references which, again, will prevent you from streaming.
YAML defines mappings to "native" elements of different other languages... but what should you do if you are parsing it in Ruby, but get a mapping to Python?
YAML schema sucks. It would require a separate "expose" to describe why.
YAML comes with no concept of users or namespaces which would be necessary for ownership / secure sharing of information. It also comes without triggers which would allow an application to respond to changes in data.
YAML is very hard to parse for no reason. It's very easy to make a typo in YAML that will not make it invalid, but will be interpreted contrary to what was intended.
YAML doesn't have a canoncial form which makes comparing two elements a non-trivial and in some cases undefined task.
YAML doesn't have variables, nor does it have forall / exists functionality. This results in a lot of tools that work with YAML overlaying it with yet another layer of configuration, which includes variables (eg. Helm charts, Ansible playbooks).
----
I mean, honestly, people who created YAML probably saw it as an inconsequential, sketchy project that should take maybe a weekend or two. They didn't plan for this format to be the best... they just made something... that sort of did something... but not really. I have no idea why something like this received the acclaim that it did.
When storing settings in config files, you can just copy the program folder to another computer and your settings will be transferred. This is called "a portable application" in the Windows world. When using the registry, you need to export the registry keys and import them, which makes it hard to backup all your apps with their settings or to set up multiple computers to use the same apps with the same settings.
[1] https://www.abareplace.com/docs/options.php#optionsStorage
The year is 2023. Can we really not come up with better things for configuration?
I mean, we did, 25 years ago With XML. The problem is that it was too much of a dumpster fire when it came to tooling, and folks just abused it until everyone was sick of it in the early naughts. But the fact remains, XML meets all the needs for configuration files, is flexible enough to handle large and small use-cases, and with the right tooling, can be a breeze to deal with.
But no. Instead we have half-solutions like json, and worse, yaml.
The hope is that as time passes, the generation that had first-hand experience with the old thing will die off and the new generation will re-discover things unjustly forgotten.
Or, another, perhaps a simpler mental experiment: try writing a CloudFormation template in an INI format. After all CloudFormation templates are configuration for the service that creates virtual appliances. If you look at the current JSON / YAML way to define this kind of template, you'll see that YAML isn't expressive enough to do it, and they had to invent a lot of meta-language conventions to be able to express what they need. With INI your struggle will be monumental to accomplish that.
One of the co-creators responded [1] to an article [2] about how Yaml is “not that great” on this site.
It comes off to me as, well we did it for free and people use it so I’m happy abuot it.
It's valid to make things w/o thorough design or deep knowledge of the domain you work in. It's a good way to learn how to do things. The blame isn't with the authors. The blame is with the community which mistook YAML for a good configuration format.
It works like the Windows registry, but now the fields also have native support for a description that tells you what the settings do.
The way I see it, Windows Registry "failed" for no reason other than a handful of CLI neckbeards refusing to let go of individual .ini and other such plain text config files written however and scattered wherever they damn well please.
The Registry is really well designed, it's a central repository of settings for everything that exists in that Windows installation including Windows itself. It can hold data besides plain text. It can be manipulated both by the CLI and GUI and is easily accessed, either individual keys or in batches. It can accomodate multi-user environments independently of the file system.
The Registry still exists and is still used by most Windows programs, though unfortunately even Windows is starting to disregard the Registry with its newer developments. The Registry for me is a reminder that the programming industry is, in many ways, one of the most regressive industries ever.
I'd say that in 70% of things I work on, part of that time is figuring out what config files are used because the app/service/process/whatever is doing something the owner/tech/whatever doesn't expect, yet is rational, and they "didn't tell it to do that".
It'd be great if everything reported as the article mentions -- amazing, in fact -- but I feel like it's a pie in the sky wish. Like asking that all programs always exit cleanly or something. I feel like there'll always be inevitable edge cases -- libraries that themselves pick up config files or registry values or environment variables that aren't expected, who knows -- that'll need to be discovered. If things are already not going right with a program, I'm less likely to trust what it says it's doing and more likely to just watch it and see what it's actually reading and doing.
I found the explanation: https://github.com/httpie/httpie/issues/480 httpie doesn't look into /usr/local/share/ca-certificates but apparently it's not httpie's fault, it's the python requests library that doesn't check that folder.
I don't know what to think of it and who is supposed to fix it. But I am back to curl with my tail between the legs (because I oversold httpie to my coworkers) for now. There's only one place where the bucket stops with curl.
> I think I forgot to run "pip3 uninstall" and ran only "pip uninstall" as I had to use python3 to get working ssl in the first place.
You can override the REQUESTS_CA_BUNDLE env var to point to your own certs.
And then document it misleadingly badly. These options are mandatory! (lists 5 of the 10 actual mandatory options).
Then a script branches on a env var value, which isn't one that's documented, and then fails opaquely if it took the wrong branch because you didn't know you needed to set that env var.
Best ones are the ones that consume your env vars and set _other_ env vars, it's great!
To highlight a few examples of how I think they're good:
> STRIMZI_ZOOKEEPER_ADMIN_SESSION_TIMEOUT_MS Optional, default 10000 ms. The session timeout for the Cluster Operator’s ZooKeeper admin client, in milliseconds. Increase the value if ZooKeeper requests from the Cluster Operator are regularly failing due to timeout issues.
We know if it's required, what it defaults, and what I love to see, why we might want to use it.
> STRIMZI_KAFKA_MIRROR_MAKER_IMAGES ... This [provided prop] is used when a KafkaMirrorMaker.spec.version property is specified but not the KafkaMirrorMaker.spec.image
I like this, explaining when this env var will or will not be used.
> STRIMZI_IMAGE_PULL_POLICY Optional. The ImagePullPolicy that is applied to containers in all pods managed by the Cluster Operator. The valid values are Always, IfNotPresent, and Never. If not specified, the Kubernetes defaults are used. Changing the policy will result in a rolling update of all your Kafka, Kafka Connect, and Kafka MirrorMaker clusters.
Firstly, always great to enumerate all accepted values for enum-like props. But what I really like here is that the consequences of altering this value are explained.
> STRIMZI_LEADER_ELECTION_IDENTITY Required when leader election is enabled. Configures the identity of a given Cluster Operator instance used during the leader election. The identity must be unique for each operator instance. You can use the downward API to configure it to the name of the pod where the Cluster Operator is deployed. (Code snippet omitted)
Interactions between config options highlighted - if you set STRIMZI_LEADER_ELECTION_ENABLED to true, this option is now required.
We're told that this must be unique. And a suggestion on one straightforward way to do this, with example code.
One more thing to call out as good:
> The environment variables are specified for the container image of the Cluster Operator in its Deployment configuration file. (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml)
Being told _where_ in the source code the env var is being consumed is great.
Now compare the Confluent docs for their Kafka Connect image. [1] A colleague of mine was using this image, and was connecting to Confluent Cloud, so needs to use an API key and secret. There's no mention of doing that at all. But you can. Just merely set the CONNECT_SASL_JAAS_CONFIG option, and make sure you also set CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM and CONNECT_SASL_MECHANISM.
And very importantly, don't forget CONNECT_SECURITY_PROTOCOL, as not only will you have odd connection failures (usually manifests as the client dying at the ApiVersions negotiation stage of the Kafka protocol's handshake), but because a script run in the image uses the value of that undocumented setting to execute a prerequisite in different ways [2], and you'll get weird behaviour that obscures the real issue.
2) Support more than one way of configuring things - maybe I want to mount in a config file, maybe I want to provide a ConfigMap, maybe I want to do it via env vars. Well... [3]
[0]: https://strimzi.io/docs/operators/latest/deploying.html#ref-...
[1]: https://docs.confluent.io/platform/current/installation/dock...
[2]: https://github.com/confluentinc/kafka-images/blob/master/kaf...
[3]: https://strimzi.io/docs/operators/latest/deploying.html#asse...
With the exception of New Age software like Go, that for some reason doesn't use manpages, I'm not convinced that the problem implied here actually exists.
If the goal is to find out which particular file is being used at this particular time, on this particular system, the maximum verbosity log might have that information, and otherwise strace can help, as described in the other comments.
The program already knows what config files it's loading and in what order, so why is it left up to the human to do all that work again? There should be a flag like --manifest or something that would spit out what actually got loaded from where.
Primarily because man pages need to go in the MANPATH to be useful but these tools tend to dwell somewhere in ~/.
So, following current practices, installing binaries in ~/.local/bin and man pages in ~/.local/share/man should work without any additional configuration.
> Ideally I'd like to avoid scanning all the way through the manual page or other documentation for the program to find out, because that's slow and annoying.
Depending on how much config you have, sooner rather than later you’ll have multiple sources: system defaults, user config files, sometimes profiles, env vars, and flags. Configuration may be sourced from a combination of these, so overrides need to be handled and naming needs to be consistent. Another source of mess is deeply nested config, such as for a user-defined project. Those are very awkward to add flags for, in which case it’s better to enforce a file (like kubernetes config).
To me, tools like git strikes a good balance. And by that I mean that project-local config is in a project root directory that’s not necessarily checked in to VCS, but picks up the config even if you’re in a subdir. It rarely causes any unwanted surprises, but at the cost of being somewhat flag heavy, which is then alleviated with user-defined aliases.
Fine, I can deal with that.
What really surprised me were a couple programs that needed to be restarted to see the updated /etc/resolv.conf as they apparently don't use glibc for DNS resolution. I used lsof to figure out what was sending DNS requests to the old name server IPs.
It turned out to be osqueryd and Apache Traffic Server.
1. There are various ways to put /etc/resolv.conf back under manual control if you don't want NM to maintain it.
Yeah, and this would be fine if both /etc/resolv.conf told you what is creating that file and NM didn't rewrite its own configuration files all the time.
Anyway, it would be not that bad if NM was kept restricted to RedHat, because the entire philosophy of that distro is that configuration files are just suggestions and the system is free to change anything at any time. But instead, that behavior infected everything.
1 - The policy of whatever configures your network last rewrites this file is spreading. Apparently systemd creates it first, and Debian has more than one network configurator for you to choose.
* loading a directory of config files, so various Configuration Management tools have easier time deploying config (especially for multi-use apps like say syslog daemon, where multiple disparate apps might want to put a part of the config in)
* a function to dry-run config validation, so CM can alert about that instead of restarting an app to a wrong config
Finding where is one has never been a problem.
strace -f <executable-file> | grep openat
https://egbert.net/blog/articles/ca-certificates-rebuild-on-...
/etc/ca-certificates.conf file
/usr/share/ca-certificates/* directory
/usr/share/ca-certificates/mozilla/* directory
/usr/share/ca-certificates// directory
/usr/local/share/ca-certificates/* directory
/usr/local/share/ca-certificates// directory
/etc/ssl/certs directory $CWD/<all-certs-read-before> files
/usr/lib/ssl/openssl.cnf file
/etc/ca-certificates/update.d directory
/etc/ca-certificates/update.d/jks-keystore directory
/etc/default/cacerts directory
/etc/java-11-openjdk/security/nss.cfg file
/usr/share/ca-certificates-java directory
/usr/lib/jvm/java-11-openjdk-amd64/lib/jvm.cfg file
/usr/share/ca-certificates-java/ca-certificates-java.jar file
/etc/ca-certificates/update.d/mono-keystore directory
/etc/mono/4.5/machine.config file
/etc/mono/assemblies/cert-sync/cert-sync.config file
/etc/mono/assemblies/Mono.Security/Mono.Security.config file
/etc/mono/assemblies/mscorlib/mscorlib.config file
/etc/mono/assemblies/System/System.config file
/etc/mono/config file
/etc/ssl/certs/ca-certificates.crt file
$HOME/.mono/config file
/usr/lib/mono/4.5/cert-sync.exe.config file
/usr/lib/mono/4.5/cert-sync.exe.config file
/usr/lib/mono/4.5/mscorlib.dll.config file
/usr/lib/mono/gac/Mono.Security/4.0.0.0_0738eb9f132ed765/Mono.Security.dll.config file
/usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll.config file
/usr/share/.mono/certs/Trust/ski-*.cer file
/usr/share/.mono/certs/new-certs/XXXXXXXX.0 file
$CWD/openssl EXECUTABLE!!! (why look in $CWD?)
/usr/bin/openssl
/usr/local/bin/openssl
/usr/local/sbin/openssl
/usr/sbin/openssl (VERY STRANGE ordering of /usr/[local/][s]bin/
debootstrap, maybe?
Also, follow the XDG base directory specification[1].
[1] https://specifications.freedesktop.org/basedir-spec/basedir-...
Granted when disk space was scarce, this reduces the ability to share dynamically linked dependencies. There are other ways around that. And in the modern world, there are some retrofitted solutions where package managers indeed do it this way. And a lot of other tools out there like Docker overlap this idea that part of what they do is disentangle the file system of unrelated programs.
But the real giga-brained answer, which most programmer-type people probably aren't emotionally ready to accept, is that not only are the ways we organize our directories wrong, but tree-based directories are just dumb in general. Everything should just go in one big directory and you can use names and tags to sort them out. Programmers are culturally obsessed with trees.
Trees and hierarchical layouts do make sense for some things, but as soon as you start symlinking, a proper graph works better - though I imagine computationally harder (a sacrifice I’m willing to make on most desktop machines tbh).
Your notion of sticking everything in a single space and using tags, searches, and filters is something I’ve wanted to explore for a while. Afaik it’s unexplored space, unless others could point me to some awesome tools or processes?
A distinction I used to make when I was teaching this stuff: on your filesystem tree, on Unix names (labels) are on the links (arrows), while on DOS/Windows names are on nodes (boxes).
If you want to explore a tag-based system, take a look at https://www.tagspaces.org/
Java will die before it gives up its practice of "let's organized code in a series of nested directories where each directory is the next qualifier in the package name"
My strongest version of this is that any application that manages any state should be able to report on that state in a human- and machine-readable way.
For instance, on my NixOS machine, the Nginx configuration is not in `/etc/nginx` but explicitly provided and can then be known with ps:
$ ps aux | grep nginx
nginx: master process /nix/store/9iiqv6c3q8zlsqfp75vd2r7f1grwbxh7-nginx-1.24.0/bin/nginx -c /nix/store/9ffljwl3nlw4rkzbyhhirycb9hjv89lr-nginx.conf
Aren't they more like global constants than variables? Loaded at startup, and never change during that run of the program. (With the exception of only explicitly being re-read on SIGUSR1 for daemon-like programs.)
And global consts, or #defines, or whatever, are things we generally don't try to avoid?
This has multiple benefits:
- You can't accidentally start Conduit with a wrong config (e.g. the config in the current working directory)
- You can have multiple Conduits running at the same time with different config
- It's easier to understand for the system administrator because there is no hidden information
Varying capabilities of different config file types is part of the reason for the variety that exists.
In Prefab, the source can be a default file, env specific defaults, live values, or machine specific overrides when doing local development.
Right now I've landed on printing out the name, value & source type and path on boot. It looks like https://gist.github.com/jdwyah/5ca4b0eacdc04414540af0761b8a0...
I think it's allright, but I'm still thinking about how to make it better. More details in: https://docs.prefab.cloud/docs/explanations/bootstrapping
https://macos-defaults.com/#%F0%9F%99%8B-what-s-a-defaults-c...
I think having a standard utility API for *nix configs makes a lot of sense. I'm surprised it doesn't exist.
I tried to find one, and there are some libraries for reading and parsing in every language, but nothing that seems to cover everything.
This bash script seems to be a fairly "built in" way to parse them: https://unix.stackexchange.com/questions/441076/which-is-the...
Exactly. But Registry has been using NTFS and it's capabilities for rollbacks and recovery. Therefore, the problem is mostly solved. There are occasions [0] of bugs causing corruptions though but they are very rare.
0. https://www.bleepingcomputer.com/news/microsoft/windows-10-n...
systemctl daemon-reload
systemctl disable mydaemon.service
ls /etc/systemd/system/mydaemon.service
ls: cannot access '/etc/systemd/system/mydaemon.service': No such file or directory
It has no business removing that symlink. It was placed there by me, and it should be treated no differently than if it was a regular file.
I have no problem with systemctl enable/disable creating and removing links in the appropriate target.d directories, but this symlink had nothing to do with that mechanism. Leave my stuff alone.
[1] https://www.freedesktop.org/software/systemd/man/systemd.uni...
> This is how symlinks have been used for configurations since forever (for example by apache on Debian).
This is not how symlinks have been used for configuration "since forever." You didn't have to add the link target to some "load path" variable when the link itself was in the correct directory.
P.S. Debian's a2ensite and a2dissite also assume that your configs are in the sites-available folder, not elsewhere in the file system.
I think it's a reasonable complaint.
I suppose you'd need to be very opinionated for it to work, but people will still bodge it to make it work with their needs that your system doesn't natively satisfy (people putting JSON strings in the values, for example).
So no, nor a washing machine nor a car ECU is expected to do this (not until sysadmins are paid to administrate them at least). As for technicians that might need to configure one someday, those people are usually expected to already know where to look, or at least to be able to read their documentation correctly. And anyway, car ECUs and washing machine usually provide a UI for configuration and don't rely on config files as a primary mode of configuration.
As for GUI programs, if they use a human readable config file under the hood, or if they accept a config file as input (some let the user change some simple settings through their UI but uses a config file to change more advanced settings), yes they should have a way to show where it is (either through a command, or through the settings UI for example).
I don't really know what you are talking about with the pages of a flash memory... Since in Unix, everything is a file, it will be read through a file anyway, not directly through pages. Same thing for offboard: just display the path to the file, wherever it might physically be. And if you really are in the most rare and hypothetical scenario where your program reads its config directly from the pages of a flash memory, just have it display "The config is read from the last [two] page[s] of <this flash memory> when invoqued with --help.
Specifically, "program" in this case refers to "CLI program that runs on Linux, used by the IT field".
(For reference, strace prints out every system call used by a program with the arguments that were called; occasionally, a program may use something other than `open` or `fopen` to read in a file, so that should be kept in mind when using this method)
Some go half-way, like HAProxy where it does spawn a new process but that process just binds with SO_REUSEPORT and signal old one to close the socket and finish all remaining connections. So you effectively get hitless reload, it's just that old process is the one finishing in-flight requests
It's a bit different for desktop apps, as restarting app every time you change some config options would be far bigger PITA than some server app.
I'm thinking of things like config for nginx, Apache, Kubernetes, etc.
USER='{
"name": "joe",
"email":"joe@example.com"
}'Config files are better, but can lead to the mess described here and typically only give us crude axes for overriding.
It's time for us to embrace & tame the complexity. If you use a feature flag tool already, you know what it's like to target different values based on rules.
If I want to experiment with changing the http timeout for specific calls from a single service in a single region I should be able to roll that out.
It's time to expand our brains and bring this feature flag style thinking to the rest of our configuration.
...why ? how often you do it ? What actual app feature it fills ? Why would you throw permanent code at something that you need to do once or twice?
That kind of thing is FAR better served as customizable hooks (loadable .so/.dll, maybe even attached Lua interpreter) rather than bazillion random flags for everything and checks cluttering code.
httpClient.get("my/url", timeout: config.get("http.timeout"))
Where config is a library smart enough to do some rule evaluation over a live updated backing store to get the right value.
I'm not sure how .so/dll or lua is going to help here.
So don't do: farblewarble too low.
Instead:
FarbleWarble is 1234 but need at least 5678. Edit /etc/whatever.conf and add: FarbleWarble="5678"
There is a special hell for programs that talk abut how the Burble is configured incorrectly , without any mention of FarbleWarble
git config --show-origin --list1. A way to print the paths to all files visible that will be read for configuration. This means that if the user config file means the system config file is ignored, print only the user config file path. Alternatively print that the system file will not be read but show its path anyways. There's many ways to solve this and it depends on the way the app loads configs.
2. A way to print the final configuration after it's been loaded and validated. This should show what settings override others. For example if the user misunderstands the config merging or erroneously quotes a numeric value causing validation to fail, this would be a great way to triangulate that.
I've seen quite a few apps implement #2 but rarely do they have #1.
I was thinking a possible naive solution to this in a file system would be some sort of pragma you could put on a file: other files it should be synced with and a date of last change. If you update a file, any file it’s supposed to be synced with must be updated, if only at the very least, the date of last change. It’s sort of a forcing function to make sure everything’s up to date that needs to be.
Could be onerous in a large system but I’m not sure.
It's a blog post where everyone ends up being slightly better after reading it.
show which sources of config data were used: files, CLI, env vars, etc.
with --verbose, also show which potential sources were not used
show any conflicts or overrides from config sources
show config settings differing from program defaults
in the same format as their source, or
in an appropriate format for a config file
On a side note, we also checked in the codegened files and that was great for other reasons like reviewability and searchability.
Who should enforce this standard? I would say community pressure does it pretty well at the moment. A standard does not need to be enforced and the XDG base directory one is still useful. The more awareness exists and the more people want this and think it is useful the better. It also has the advantage that no other competing standard exists as far as I know but it still has some downsides.
more and more im feeling software deserves to be dumped into a nix style sandbox if it refuses to behave properly. I've had to recompile shit from source just to get it to respect my environment variables regarding where to look for local configs.
[1] https://www.cisa.gov/sbom#:~:text=A%20%E2%80%9Csoftware%20bi....
like if you need LLVM and didnt manage to find it under
"app_folder/llvm"
then let's go find it somewhere on the disk(s) and ask user if he wants to use it
Python has platformdirs for this. It handles XDG and a number of corner cases.
https://pypi.org/project/platformdirs/
Ideally the OS you deploy to should handle this as well.
You can make a actual API to expose application configuration knobs too which is nice but you should tie this into your rollout/CICD system so you're just as careful pushing new configuration values as you are when pushing new binaries (multiple envs, slow rollouts, blue/green etc).
Be smarter about package management and this is a non-issue; checksums, install-time dumps of file modifications, and the like.
It’s why tools like CFEngine and the rest were invented, and I’m upset this is still an issue for people like OP twenty years later.
I don’t think anyone ever used that function.
Other way is to run program in symbomlic interpreter and trace what it wants to read in any of the branches.
Why do you suggest this path over documentating the behavior explicitly?
What makes one approach better the other? Is it faster to run a trace? Should a developer trace their own code and auto-generate documentation post-build?
Like helm chart definitions. Now there is an upgrade of the chart, and I need to go and trace down which value changed.
Well, if we would've kept the commented stuff I could overlay it with the new defaults showing me where the defaults have changed. At least that gives me some clue.
If I'm only setting `persistence.storageClass` and `ingress.host`, my values.yaml will be two lines, not the original 2,000.
* Config files
* CLI options
* Environment variables
And what takes precedence when.
Now, when I run `cmd -h`, I get output like the following for all flags (example from [1]):
--count , -c : Number of times to dig
type : int
default : 1
configpath : dig.combine.count
required : true
currentvalue (set by config) : 1
[0] https://github.com/bbkane/warg [1] https://github.com/bbkane/shovel
It would have saved me time today when re-visiting and extending a service that I’ve not touched in two years.
The operating system should tell systemd where to put them.
The operating system does not use JSON.
Users can then ask the operating system.
Makes life a lot easier imho.
I don't get to write Scala anymore and now I really miss it for certain things.
Really enhances the debugging experience.
That might be useful.
I have trouble seeing the rationale for other locations when the XDG standard exists
It allows you to trace system calls, and, crucially, file opens.
if you know the PID aka Process ID for the application yr running try lsof -p PID_NUM_HERE
it will give you a list of every open file your application is using along with it's full path
finally, if that gave you too many lines try lsof -p PID_NUM_HERE | grep -E 'txt|properties|config'
I know you wanted the files listed it in a help file however this will work 100% of the time
Best, Stephen
I know how we get here, people simply report the error from the OS w/out adding context. This is why I love anyhow reporting in Rust, you can attach context (like the file name and path) to the error.
In log-store.com I report the default name of the config file (you can specify it via cmdline if you want), and the 3 locations and order searched, if the config file isn't found. I also have it report the file being used, so if you expected the file in /etc, but you accidentally had one in your home directory, you'll know on startup.
These things seem like table-stakes in 2023.
cat /var/lib/dpkg/info/<package>.conffiles
?Luckily ctrl-r helps the second time.
It provides a ready-to-use --config option which reports how it sources the configuration file [2]:
$ my-cli --verbosity DEBUG subcommand
debug: Load configuration matching ~/.config/my-cli/*.{toml,yaml,yml,json,ini,xml}
debug: Pattern is not an URL.
debug: Search local file system.
debug: No configuration file found.
(...)
$ cli --int-param1 3 --show-params
╭─────────────────┬─────────────────────────────────────────┬─────────────────────────────────────────┬──────┬──────────────────┬─────────┬─────────────────┬─────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────┬─────────────╮
│ ID │ Class │ Spec. │ Type │ Allowed in conf? │ Exposed │ Env. vars. │ Default │ Value │ Source │
├─────────────────┼─────────────────────────────────────────┼─────────────────────────────────────────┼──────┼──────────────────┼─────────┼─────────────────┼─────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼─────────────┤
│ cli.color │ click_extra.colorize.ColorOption │ --color, --ansi / --no-color, --no-ansi │ bool │ │ │ CLI_COLOR │ True │ True │ DEFAULT │
│ cli.config │ click_extra.config.ConfigOption │ -C, --config CONFIG_PATH │ str │ │ │ CLI_CONFIG │ /home/runner/.config/cli/*.{toml,yaml,yml,json,ini,xml} │ /home/runner/.config/cli/*.{toml,yaml,yml,json,ini,xml} │ DEFAULT │
│ cli.help │ click_extra.colorize.HelpOption │ -h, --help │ bool │ │ │ CLI_HELP │ False │ False │ DEFAULT │
│ cli.int_param1 │ cloup._params.Option │ --int-param1 INTEGER │ int │ │ │ CLI_INT_PARAM1 │ 10 │ 3 │ COMMANDLINE │
│ cli.int_param2 │ cloup._params.Option │ --int-param2 INTEGER │ int │ │ │ CLI_INT_PARAM2 │ 555 │ 555 │ DEFAULT │
│ cli.show_params │ click_extra.parameters.ShowParamsOption │ --show-params │ bool │ │ │ CLI_SHOW_PARAMS │ False │ True │ COMMANDLINE │
│ cli.time │ click_extra.timer.TimerOption │ --time / --no-time │ bool │ │ │ CLI_TIME │ False │ False │ DEFAULT │
│ cli.verbosity │ click_extra.logging.VerbosityOption │ -v, --verbosity LEVEL │ str │ │ │ CLI_VERBOSITY │ WARNING │ Debug │ COMMANDLINE │
│ cli.version │ click_extra.version.VersionOption │ --version │ bool │ │ │ CLI_VERSION │ False │ False │ DEFAULT │
╰─────────────────┴─────────────────────────────────────────┴─────────────────────────────────────────┴──────┴──────────────────┴─────────┴─────────────────┴─────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────┴─────────────╯
[2]: https://kdeldycke.github.io/click-extra/config.html
[3]: https://kdeldycke.github.io/click-extra/parameters.html#show...
Lsof, find files open on running proc?
Problem solved?
Especially those out-of-context .yaml files with lots of DSLs embedded.