Dotfiles being hidden is a UNIXv2 mistake (2012)
web.archive.org
web.archive.org
"In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move"
The Universe is the interior of the Light Cone of the Creation
Science is a Differential Equation. Religion is a Boundary Condition
- Alan Turing
And the origins of /usr/bin a definite third.
Honest question: How would you design a simple text outputting protocol? Would it be radically different to how it's done today, or would it be the same basic ideas but cleaned up a bit?
The one downside that has is if you separate the formatting from the data you can not recreate the session without keeping time info somehow. So, might need a fix for that.
The only other real alternative I'm aware of is moving the formatting into the API "stream" that communicates using the machine ABI. This is a terrible solution. It's what Windows uses.
I think the biggest danger would be scope creep, but a terminal shouldn’t just be thought of as text.
We already have gtk, qt, appkit, win32, web based frontends for almost everything, and none of them produced anything remotely interoperable. The main advantage of just text is that you see its structure right there in a terminal and may reason about it without referring to the documentation of ps result-related set of structures with all the possible bells and whistles embedded into a large xml-like document schema (which is of course full of legacy and references to other specs, because compatibility and standardization).
You can program this message in the setup screen of the terminal. Maybe you can set up all the terminals in the computer center at night...
You can send Ctrl-E to a user with "write" or "wall"..
This answerback message is actually really old, exists even in mechanical ASR-33 teletypes:
https://en.wikipedia.org/wiki/Enquiry_character
It can also be triggered by the mysterious "here is" key on the ADM-3A.
One would hope that we're past the bad times, but at the same time brand new projects like the kitty terminal emulator let stdout read and delete arbitrary files. And verifying something is safe is just hard, these are not even e.g. memory safety bugs, they're "features".
Oh come on, you know what everyone means. "Output with potentially hostile content".
Right now it seems kitty will happily delete e.g. /tmp/.X11-unix/X0. That sounds fun.
https://sw.kovidgoyal.net/kitty/graphics-protocol/#the-trans...
https://github.com/kovidgoyal/kitty/blob/5541e3c2ff441aea319...
> And no unlike your claim, it does not read them.
strace -e open,openat kitty
printf '\e_Gf=24,s=10,v=20,t=f;%s\e\\' "$(echo -n /etc/passwd|base64)"
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 14Me asking my editor to open /etc/passwd is me asking my editor to do something. My terminal emulator opening files and parsing them and deleting them just because I viewed untrusted content is something different. If you want to make that argument, you need to come up with an example where my editor accesses attacker-controlled files because I opened a random README.md.
Also, please read https://news.ycombinator.com/newsguidelines.html, and shove your attitude.
You very conveniently left out the fact that pretty much anything that views untrusted content has to parse files. If parsing files is where your "security boundaries" lie, I suggest you drop your computer off at the garbage dump. Hell if your editor has a preview function it too will parse untrusted content. Basically, to do anything software has to parse untrusted content. And just by the way, /etc/passwd is not attacker controlled. If the attacker has access to your filesystem already, she really doesnt need to use kitty to do anything. And "opening" README.md will not cause kitty to parse files, unless by "opening" you mean catting without -v. In which case we are back to your mommy told you to know better but you didn't listen.
At this point its obvious you are deliberately trying to spread FUD. I am done interacting with you. Good bye.
The general trick with parsing is to put it in a sandbox that cannot touch files, can only do limited syscalls, etc. This is what e.g. Chrome does.
But yes, it's obvious the Kitty author has no interest in making it more secure ( https://github.com/kovidgoyal/kitty/issues/2084) and it's obviously you take discussions about improving software security as attacks on your person.
Have a good day, sir.
Furthermore the next release of kitty has capability based security for remote control with public key crypto to keep the data safe: https://github.com/kovidgoyal/kitty/discussions/5320
I suggest you find a better place to move the goalposts in your attempts to smear and spread FUD.
If I were to design terminal control today, I might have a separate file descriptor for a pipe that is used for control communication instead of sending control codes inline with the text. And I would definitely have a way to query the capabilities of the terminal directly, instead of looking it up in a terminfo database.
How do you keep that synchronized with the text for things like moving the cursor between words?
r-chive.org
-- from Wikipedia: https://en.wikipedia.org/wiki/MOS_Technology_6581
The VI keyboard shortcuts only make logical sense on a specific Tektronix 4014 terminal that was popular at the time: 46 years ago.
The reason the delete key is broken in like half of all UNIX / Linux systems is because TTY is short for “teletypewriter” — literally a remote type writer banging out text on dead trees. The carriage can go back one character at a time and overwrite an error but it can’t shift printed text on paper.
Windows was developed in an era of CRT displays and delete works with it consistently.
This is why it cracks me up whenever someone talks a out “Linux being the standard”…
The Lear Siegler ADM-3A, actually:
http://www.fabglib.org/group___enumerations_ga67edd9a04814c2...
The Windows Terminal team copied the UNIX terminal "standard" of only clearing the current viewport.
Now, scrolling back "works" in the sense that it shows partial(!) output from previous commands.
Say you have a script that produces 47 screens worth of output, and what you want to see is somewhere in the middle. You can't judge this visually because the scrollbar is inaccurate and there may be other history.
Before you could just 'cls', re-run the script after a change, scroll back, and see only the updated output.
Now, if you do this you'll get 46 screens of superficially identical looking output, with 47 of the output you wanted to see appended to it. This is super confusing.
It has confused me. I've seen other admins be confused as well and copy-paste the wrong thing.
The Microsoft Terminal team debated this and decided to do the wrong thing on purpose and copy the "Linux way" to pander to Linux admins on Windows.
Meanwhile if you go back through the old newsgroups and mailing lists, the same argument was had by Linux people as well, with the same logic I just outlined. They also decided to do the wrong thing. Why? Because they were pandering to UNIX admins coming across to Linux.
Why does UNIX do this? Because of an error in the coding. That is all.
That's why the shiny new Windows Terminal is broken now, to copy an error that was already a copy of an error.
Busybox clear does not clear scrollback nor does it provide an option to do it. But for example on Alpine where Busybox clear is the default, the ncurses package can be installed to get the ncurses clear.
Microsot teams are tainting the Windows developer experience trying to cater to the crowd that has been buying Apple gear for GNU/Linux development.
Are you saying this specifically or metaphorically? Because I suspect that there was a rationale behind that decision.
See also https://man7.org/linux/man-pages/man1/clear.1.html which explicitly states that it clears scrollback iff a terminal has a specified capability. Which windows terminal being of CRT origin should have. I think the issue here is actually an intersection of some terminal nuance and windows team who blindly copied the behavior that some linux guys were used to because this knowledge was buried under their gnome defaults.
Iow, no need to blame unix and linux, blame those who picked one (stupid) mode of many and used that as the only mode of operation.
And why it makes more sense for \n to be the common Windows line ending because it doesn't have that TTY ancestry.
while true; do echo -n " $(date)\r"; sleep 1; done while true; do printf " $(date)\r"; sleep 1; done printf '\033c'; while true; do echo -en " $(date)\r"; sleep 1; done while true; do echo -en " $(date)\r"; sleep 1; doneAnd xpg_echo is on by default for bash on macOS, so for users on macOS it looks like echo interprets escapes by default.
This is one of the reasons why echo is hopelessly unportable.
And I wish we had a stronger word than unportable for echo. You see that word tossed around on shell scripting resources where the only system it applies to is a 30 year old version of ksh and it only blows up on a full moon when passed a filename with a literal form feed \f, but echo is a practical portability problem.
vi is more than hjkl, and those four non-mnemonic keybindings make up for it by being extremely easy to press for QWERTY users. The vast majority of other commands are perfectly “reasonable” (easy to memorize and not dependent on any antiquated hardware) — append, backward word, change, delete, end of word, find letter, go to (OK, this one is just an arbitrary prefix), and so forth.
Which Linux system/terminal/editor are you using where insert mode is not the default?
Not many? In the sense that, considering several UNIX and C design issues and footguns, the dotfiles are the least of our worries (compare e.g. to C buffer overflows or shell escaping rules, to name but two)...
I also don't particularly like the repeated ditches at "lazy programmers" - isn't the whole "New Jersey" style [1] heavy on laziness?
In fact, if those "lazy programmers" were given OS level-APIs and libs doing the right thing about dotfiles, they wouldn't have recoded them on their own in their programs...
[1] https://en.wikipedia.org/wiki/Worse_is_better#New_Jersey_sty...
In terms of wasted CPU cycles, it's another story but hey these are the days of using npm packages to basically do a string comparison.
I have some rough thoughts on expressing why:
> This leads one of the major cognitive biases in software development where people speak of tradeoffs without establishing they are Pareto Optimal, and wrongly arriving at the conclusion “nothing more can be done without a huge effort”.
https://www.paretooptimal.dev/pareto-optimality-and-software...
That's ironic given your handle on HN :)
>This leads one of the major cognitive biases in software development where people speak of tradeoffs without establishing they are Pareto Optimal, and wrongly arriving at the conclusion “nothing more can be done without a huge effort”.
Well, part of the idea is to do the "pareto optimal" work not just implementation-wise, but also in checking whether the tradeoffs are pareto optimal (pareto-ception). Thus, avoid "analysis paralysis"
This is just not true. Software updates break things all the time. Having software updated behind your back is incredibly inconvenient.
Is there a way to improve testing while gaining the benefits of auto updates?
The problem now is how to make it safer
So software has bugs, and sometimes those bugs are bad, so we want to do something to mitigate that risk -- the risk that we might have something as serious as RCE in our software.
To reduce the chances of somebody exploiting that hypothetical remote code execution vulnerability we have special code running on our machines that reaches out to download new code and executes it, blindly trusting that every Jo Schmo who's ever written software we were comfortable using once would never take advantage of that position.
The fact that automatic updates sometimes have some benefits right now seems almost accidental when their fundamental operating principle is to open exactly the same sort of backdoor they're trying to protect against.
Is this always the case? Nope. But for most cases, updating often against newly exposed threats is worth the vulnerability that auto update introduces.
But like with every solution to a problem, you have to decide what’s best for you, your users, and the internet as a whole. Quite the balancing act.
- 72h continuous work hours later
Why is everyone looking suicidal?
Absolutely Disgusting.
! name[1 + (name[1] == '.')]
https://en.m.wikipedia.org/wiki/X86_instruction_listings
For example, JSR becomes pretty self explanatory when you know it stands for Jump To Subroutine.
The complexity is more in that you have to do many simple tasks on your own. It may take 50 lines of code in Assembly to do something that Java can do in one line of code.
And, especially while reading it, you have to keep track of a lot of the program state in your mind yourself, and that gets messy when you go beyond small, single-purpose chunks of code.
I love dotfiles. When I list a folder, I don’t want to see all the cruft, I want to see the key stuff. Sometimes when I organizing something new, I use .directoryName to hide things temporarily.
git is one of my favourite examples: .git hides stuff I do NOT need to see every day, leaving behind the things I am actually working on, the things I care about.
Besides, it is so easy, once one has CLIed a while, to find (no pun intended) and/or account for dot files when you need to.
If you hide something (like a folder) temporarily and refer to it through a script, you have to go to the script and change it once you decide that you don't want the folder hidden anymore. The script is now coupled to not only the location of the resource, but also its visibility.
There is no reason to conflate the two concepts. Git folders can still be hidden by default without requiring adaptation to path names.
I'd argue, what you're loving about dotfiles is not the presence of the dot, but the concept of visibility levels on files.
The only way to get truly hidden folders is to agree on an approach and have it adopted everywhere. Someone’s git-aware GUI app might know to hide git, but as a CLI guy, should I have ls modified? Or find? Or my shell (if so, which one)?
The . prefix hides things most users don’t need to see (. and ..) and allows many other things to be usefully hidden in a convenient and consistent manner.
To rephrase my original comment and integrate the crux of yours, Pike is wrong to call this a mistake, because it accidentally gave us a really useful capability that has been widely integrated into most/all of our tools, while allowing us to work around it simply and effectively when we want or need to.
Sorry, not trying to knock your idea, just had to let my inner pedant out a bit.
Within the UNIX/Linux world, though, it’s a remarkably useful and effective convention, implemented and respected with considerable consistency.
(Windows hidden files really aren’t, not once one drops to a CLI, and even then, it depends on the CLI. I’m not offering that as a criticism, but more of a caution of how difficult it is to get something like this to work. Extended attributes and even basic permissions have their twists and difficulties, even within a platform or platform family, let alone cross platform.)
this should have been a concern from the beginning. instead of the hackish workaround of encoding attributes into the filename. they should never have been there. even file-type extensions should have been a separate field of some sort right from the start.
Or like magic numbers in UNIX/Linux?
The whole extension thing was odd from the beginning. When I first used Windows, after years of UNIX, extensions puzzled me. On UNIX, they told ME something, the OS and apps didn’t need them (/etc/magic did the trick), but here they were on Windows being given semantic value.
Ironically, now that I think about it, I found Windows extensions odd for the same reason that some find hiding dotfiles on UNIX/Linux odd. This makes me chuckle ruefully.
magic numbers are a crude, expensive, and not always working workaround to detect the type of a file based on the first bytes of a data-stream. and there are programs that refuse to load a file if it has an extension that is not supported, even if the content of the file would be just fine. (can't think of an example though, but i definitely remember getting irritated when that happened)
what bothered me about extensions in unix is that they were sometimes arbitrary or wrong. especially on the web, where urls had arbitrary extensions that related to how the page was produced, and not the content that was served (which was almost always html. other extensions only matched for files to be downloaded, but you could never trust it, so in most cases it served no purpose)
today i would probably use extended attributes, since they are a lot more flexible
> that's unnecessarily surprising.
I found it unnecessarily surprising that dotfiles werent shown when I got started with unix-like systems. I consider "surprise" to be a weak argument in this case, especially since we do have things that refer to a file that both the file browser and ls do not show (but which are still loaded since opendir and readdir do not care about the dot).
Whether or not a file with a dot at the beginning is shown or not is nothing but arbitrary convention. I'd like my convention to not care about the file path to decide "does ls or the file browser show it", as that is how I prefer different concepts in general: decoupled.
However, I do accept surprise as an argument against hidden files at all (including dotfiles).
That is highly implausible, except as a case of extremely distracted coding.
Someone like Thompson or Ritchie wouldn't consciously write a test for just the leading dot without being fully aware that it matches more than . and .. .
Even if the thought had been "nobody in their right mind will have files starting with dot, so who cares", that would still make it a deliberate requirement being consciously implemented and not a mistake.
By distracted coding I mean that situations like this happen: you intend to write a piece of code which does something specific. The code is motivated by certain happy cases. You start coding it and get it into a state in which it runs but doesn't quite do what was intended. Then you get completely distracted by some interruption. You forget that the code hadn't been completed; you mistake the remembered intent for the completion. You run the code and it carries out the intent on the motivating happy cases, and somehow fool yourself into believing it was done.
Maybe Pike meant that it was a mistake to create that requirement.
I don't hate the $HOME/cfg directory idea, but please call it $HOME/.cfg so I don't have to see it. :)
Why not? Alternatively, who said the decision was 'conscious'? It doesn't take a dumb person to make a dumb mistake. Brilliant people get things wrong all the time, and anyone can be hasty or careless in a moment.
It's not a coding mistake where some wrong identifier was used or other typo; coding the proper test requires more instructions. You have to check that there is either a 0 byte after the first '.', or else another '.' byte that is followed by a 0.
Here is how someone could be distracted halfway through and forget to finish it. Suppose the code has a structure like this:
for (;;) {
if (getnext(name) < 0)
break;
/* code added to skip . and .. */
if (name[0] != '.') /* if it doesn't start with a dot, print it */
goto print;
/* This can be continued here with more logic without changing the above line;
But it would already skip . and .. with just the above line. */
if (name[1]) { /* two or more chars */
if (name[1] != '.') /* Dot followed by non-dot: print it */
goto print;
if (name[2] != 0) /* Two dots, followed by non-dot: print it */
goto print;
/* confirmed dot-dot */
}
/* confirmed dot or dot-dot */
continue;
print:
puts(name);
}
cmpb (r3),$'.
bne 2f
sub $8.,r1
br 1b
2:
jsr r5,gstat
br 1b
If the behavior was indeed not intended, this would be a case of incomplete code. Incomplete code can happen by distraction. You turn your attention to something else, perhaps for the span of several days, and then don't remember the incomplete state. The brain had visualized the complete state, leaving a false memory of that having been done.
On a related note, one element that is gapingly absent from all historic Unix sources is ... test cases!
Compiler test cases, shell test cases, utility test cases, library function (and system call) test cases ...
They are nowhere to be seen.
There is something nice and immediate about adding a dot to make it invisible to certain things (ls by default, the file explorers in ubuntu by default, etc.)
A bit like file extensions themselves and the weight they carry in Windows.
It sorts of balances in favour of immediacy of the function over "right way to do things", where the right way might be some kind of meta data for the file, or for it's folder.
Funny thing about hiding, is what does it mean to hide? If you can ask your utility to show hidden things. It sort of means "things I reckon people don't wanna see, or shouldn't touch, unless they are willing to take a step to see it".
A bit like a shibboleth where if you can't figure out how to see it, you are too dangerous/dumb to edit it, or even know about it.
You could instead have security settings for that. Maybe you just don't give the user access to the files at all until they elevate from user to "poweruser". Which would be somewhere between user and superuser. Use chmod to set the permissions, instead of a dot filename format.
He said the decision was certainly a mistake, which is an error in judgement, and pretty sure it was an accident, which is an unexpected incident. It is a mistake to conflate the two.
One thing I like about hidden files via the dot prefix is that I feel like it makes clear that the files are hidden only by convention. It's not a security feature, it's just a convenience to help spare you the mental overhead of witnessing the clutter.
I've gone so far as to keep my config directory visible -- ~/.config is a symlink to ~/config, which actually holds all my config, conveniently a git repository in itself for easy portability and tracking.
Things like dconf and about:config exists in no man's land, where neither generic GUI skills nor generic unixy skills are applicable.
How is this a problem with dconf? GNOME Settings for example is typical GUI preferences and works as a front-end to the dconf database.
Edit: Although I agree that some situations may require a readable text format, like a dotfiles repo. I personally use the `dconf-editor` program to find the right keys and the `dconf load` command to sync them from a regedit-like file format
Neither of these is a complete answer, but
1. Distros like NixOS and GuixSD address this successfully by emitting the config files themselves. They don't statefully edit your configs but instead store them in their own language and then translate them as wholes.
2. You can mitigate much of the pain of this with something like etckeeper, which essentially gives you version control for /etc.
I don't care. They can use OSes designed for their needs, like macOS and Windows.
Not sure why there is such a persistent idea that "everyone using Linux, including entirely non-technical users who just want an appliance" is actually a desirable or realistic goal.
And the funny thing is that everyone does use Unix. Everyone with a smartphone does as both iOS and Android are derivatives of Unix-OSes (Darwin and Linux). Only Fuchsia will be truly non Unix. If it ever makes it out the door.
However those mobile OSes (or ChromeOS) are of course not what we're talking about when we say we love Unix. The same will happen if Linux ever makes it mainstream. Consumers might love it but we won't. I'm happy for it to stay a niche thing. My niche.
You're responding to complaints that settings done this way are user-hostile, opaque, and require a ton of insider knowledge to even find. So if the only goal is user-friendliness, we're moving away from it, not towards it.
(The answer is not to type "sound" into the search bar. That takes you to the "new" sound control panel, which doesn't have all the old features, including disabling sound devices. So you have to open the control panel, go to Devices & Sound, and then you can search around to disable a device.)
The fundamental difference between Windows and Linux here is that if you write an instructional blog post to fix this problem for Windows, you can put some drive-by malware installer on it and make some money. You could obviously malware up similar Linux documentation, but by the time you get discovered you will have only infected 1 person, simply because it's much less widely used. Unfortunate!
I had to check literally more than a dozen locations in the registry (some in different hives) just to see where the thing was actually set to autoload. I then had to create new keys in still another location whose type was simply arbitrary binary data, and the only way to determine what actual value was needed there was to examine the registry before and after disabling and then reenabling the startup profile in the GUI.
The Windows registry is an absolute fucking shitshow, and it's a big part of the reason that the OS is inherently hostile to automation. The inherent invisibility of the registry and the systematic underdocumentation of registry schemas for applications and OS components makes figuring out how to do anything in a repeatable way a fucking quest.
> I personally use the `dconf-editor` program to find the right keys
For GUI applications, config files are often undocumented anyway. But to me having to run an application and see how it messes with its configuration in some database when you click buttons is an insane way to figure out how settings are declared or stored. I would so much rather the configuration format be plaintext and documented than be expected to reverse engineer my apps' friggin' config files!
The thing we want to optimize for is "how easy is to change a setting". Firefox's "about:" seems like a reasonable approach for expert users.
You wouldn't even need that many basic types to cover about 90%+ of the state space. Of course, there would be bonus points if applications could define their own types in a UI-meaningful way.
These issues neither disappear nor improve when you hide the settings in a mystery binary.
It would be nice to have a base level of types that all programs could agree on.
Same problem with Firefox; hamburger->settings will take you to a GUI that contains maybe 0.01% of all the settings Firefox actually has. You have to be "in the know" to know that about:config exists. I think the first time most firefox users find out about about:config is when they're angrily searching the web for a solution to their problems, which is basically how I discovered dconf-editor too. You don't find either of these by searching through the GUIs like a normal computer user, nor will you find them by poking around in your dotfiles like a seasoned console jockey.
Any setting in about:config can be set in user.js and your user.js file can be diffed, grepped, version controlled, edited with your favorite editor, and copied between profiles/machines.
Sadly, Mozilla is moving a lot of config out of about:config/user.js and into multiple random sqlite files, so sane config management is getting harder with each new release.
If you want to grep everything in a sqlite database, then ".dump" it.
$ echo "create table myconfig(param text, value text);
insert into myconfig values('scrollbar', 'wide');
insert into myconfig values('verbose', 'no');" | sqlite3 myconfig.db
$ sqlite3 myconfig.db '.dump myconfig' | grep verbose
INSERT INTO "myconfig" VALUES('verbose','no');Oh and we might as well store these config file in a different place on each distro and move them around once in a while just to keep things interesting. Users can ignore upstream documentation and just strace the process if they wanna know where the files are this year.
> Grep is a huge part of my workflow.
You can flatten and grep structured binary formats too.
# ll /lib64/libc-2.17.so /lib64/libsqlite3.so.0.8.6
-rwxr-xr-x. 1 root root 2156664 May 18 20:27 /lib64/libc-2.17.so
-rwxr-xr-x. 1 root root 753272 Jan 27 2020 /lib64/libsqlite3.so.0.8.6Manticore vs Elasticsearch startup time benchmark - https://manticoresearch.com/blog/manticore-alternative-to-el...
Also, not all config really lends itself to rows in a database (or some other binary registry). For example, commands to run on login. Or really anything where order matters, would be a pain to edit.
I'm not quite sure why this is a good analogy, but it feels like it is.
Thanks, but I’ll just deal with a bunch of dotfiles. This sounds an awful lot like a “and now you have two problems” situation.
It's like driving a car instead of walking is trading one problem for two.
I'm to blame actually, it does a lot more than config management, so I mischaracterized it myself.
I’ll even forgo having my own hidden files there, I don’t care. It’s so annoying.
ln -sg '~/config/*' '~/.*'Sometimes I wonder how much defaulting to a directory with a leading dot hurt adoption. It gave me the impression that it was designed for or by people who don't manually edit their config files.
Also, if you know how to change config of an application for the last 15-16 years, you also know that you're going to edit a "dot file".
KDE stored everything under ".kde" for a long long time, and stored a great tree under it too. rc files as config files and other KDE related files as app specific files.
In fact, the ".config" folder is just an evolution of ".kde" folder. Spec is written by KDE guys.
By default I'm referring to the following part of the spec and the fact that systems I've used start in this state:
> If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.
Also, some applications use their older files if they find it, and re-create under XDG spec when they fail to find that file. KDE did that to great extent and success, for example.
tools that respect XDG, for fellow JS CLI developers:
- https://github.com/davidtheclark/cosmiconfig Find and load configuration from a package.json property, rc file, or CommonJS module. [Check `searchPaths` to implement XDG spec compliance.](https://github.com/davidtheclark/cosmiconfig/issues/152)
- Sindre's libraries use [`env-paths`](https://github.com/sindresorhus/env-paths#pathsconfig) to get paths compliant with this.
- https://github.com/sindresorhus/conf simple config storing (maybe try [conf-cli](https://github.com/natzcam/conf-cli) to manipulate if needed) the successor to [configstore](https://github.com/sindresorhus/conf#how-is-this-different-f...)
- https://github.com/jonschlinkert/data-store conf like datastore but in the shclinkerverse
That does sound annoying; if only every application would just use ~/.config/appname then your problem would be solved.
I'm confused by your double negative. It looks like you're saying CLI tools should follow XDG, but the tone of your post sounds like the opposite.
If the latter, what are you, some of kind of monster? :) If you don't want a specific XDG config dir, set it to ~. Don't force us all to follow some antiquated convention from days of yore.
I interpret this as "file systems need to have a flexible metadata mechanism instead of ad-hoc hacks". Of course, at this point in time no such thing will happen, because it's easier to invent 100 new databases than change some assumptions about our core architecture.
What is quite interesting, because it's all a matter of user interface. It's a change that doesn't even impact on software compatibility. Any distro that decides that a file is hidden due to an attr can just push that change, and nothing will break.
Pike isn't even talking about that, only the typical config files in the home dir that clutter and affect performance of file name ops in the home dir.
> (For those who object that dot files serve a purpose, I don't dispute that but counter that it's the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.)
I have to agree. I can't think of any situation where hiding really helps. If the reason is clutter, the solution should be a dir.
The irony of dot-files is that they became used for metadata, like .git directories, .htcasses files and so on. So, a solution that fakes metadata by abusing names is now used to indicate metadata at another level of the system. There is clearly something missing in file system design.
I can see the use-case of "when zipping or sending a dir to a different machine, omit the metadata", but this doesn't hold true for code repos, and is sometimes even the opposite of what you want. It'd probably be better with dir name conventions like "cache", "secrets", "config" or similar.
Should .git directory reside in the same directory as your source code? Maybe not. Fossil has an interesting compromise in that regard.
Should compiled artefacts reside in the same directory as your source code? Almost certainly not. This seems like a bad idea from some hacky implementation decades ago.
Should tools be able to distinguish between "real" files and metadata files? Almost certainly yes, except there are different types of "metadata" files, as you pointed out. I think naming conventions are simply not good enough to handle this properly. This is where FS metadata could come into play.
Generally, I think if someone without prejudice examined the common problems we have with data these days and designed a "file" system from scratch, they would end up with something vastly different from our current files/folders paradigm.
Indeed, many dependency managers (both go mod and cargo iirc) don't vendor deps into your working dir.
> I think naming conventions are simply not good enough to handle this properly. This is where FS metadata could come into play.
Yeah, the ideal solution may be just that, but I'm not 100% convinced that dirname conventions and tooling can't solve the major hurdles. New file systems would take decades even if they were to take on.. And for VCS and other tools you need to account for many different file systems and can't rely on anything but largest-common-denominator features..
xattr's require a system call for each path you want to look up, and then another system call for each value you actually want to retrieve. Plus all the error handling that goes along with those calls. Oh, and they're all strings, so even more work if you want to have an integer attribute. Also, which quota should the space used by the attributes accrue to? Should we split that into "user" and "system" attributes? Whoops, looks like we need an "ad-hoc" namespace here too... this time with actual security implications!
The mechanism exists, but it's more cumbersome than it's worth. That's certainly the way I've felt about it every time I have to muck around with a system that "mistakenly" decided to use them instead of just a database.
In that light, the dot is not a historical mistake. It's an obvious optimization of a particularly popular use case.
Is it what really happened (lots of programmers, by mistake, wrote a comparison function which is just plain wrong), or Rob Pike is conjecturing?
It's somewhat easy to see how someone would make this shortcut when programming in C, but they weren't programming in C, they were programming in assembly (as the article describes). It's so much simpler to only check the first char when programming in assembly, so it's easier to see why the shortcut might've been taken.
For all intents and purposes at the time, it was a hack but it worked, and it worked well.
There was also a lot less code in general so I would expect every line to be considered in much more detail. Also, 'laziness' tends to be a character flaw that manifests itself everywhere in a person's work, not just in one place. And Richie/Kernighan don't exactly have a bad reputation.
I wonder if the '-a' parameter of ls appeared at the same time. If so it definitely wasn't an oversight.
In assembler `name[0]=='.'` is simple to write. Fetch what name points to and compare it to a literal `'.'`.
A function call on the other hand requires pushing registers to the stack, pushing the arguments to the stack, call the function, and then restoring the registers.
So if you are in a hurry and just want to try something, you will write the equivalent to `name[0]=='.'`.
I think hiding files with a path convention is Good, Actually, but I'm comfortable being in the minority on that, and it's possible that if we instead had a hidden flag a la chmod permissions I would prefer that.
With "hidden" being a file attribute (like on MS-DOS), you'd have the risk of a filename collision with a file that you don't know is there.
I like to keep an eye on the costs of what code I'm writing to make sure I don't get too 21st century and let my codebase bloat until it's slow as molasses, but at the same time I happily harvest the benefits of usually not caring whether this function call is inlined or not, or caring about the exact number of cycles an indirect function call takes, etc. It's a delicate balance but worthwhile to cultivate it.
> It was in assembler then, but the code in question was equivalent to something like this:
> if (name[0] == '.') continue;
> This statement was a little shorter than what it should have been, which is
> if (strcmp(name, ".") == 0 || strcmp(name, "..") == 0) continue;
There's no strcmp() in the first example.
if (name[0] == '.') {
if (name[1] == 0) continue;
if (name[1] == '.' && name[2] == 0) continue;
}"The unexpected appearance of an interpreter tended to freeze the form of the language, and some of the decisions made rather lightheartedly for the ``Recursive functions ...'' paper later proved unfortunate. These included the COND notation for conditional expressions which leads to an unnecessary depth of parentheses, and the use of the number zero to denote the empty list NIL and the truth value false. Besides encouraging pornographic programming, giving a special interpretation to the address 0 has caused difficulties in all subsequent implementations. " (History of Lisp, 1979)
https://stackoverflow.com/questions/8547142/what-did-john-mc...
> 0==() has been the emoticon for pornography since 1958.
> The fact that too many implementation details were leaking at a higher level, i.e. showing up too much
> Code that uses intimate knowledge.
Nothing against software efficiency, but those hundred loop iterations checking a string aren't going to be the big time-waster of your daily life.
This leaves me to wonder, what was the structure early on?
I don't know how the original Unix was, but presumably flat.
Part of which is making sure that the defaults the program chooses may be overridden by consistent environment variables.
Programs need some way to persist their own state and configuration, after all.
The fact that XDG is more honored in the breach is tiresome, but this can at least be corrected, and it's well enough put together to be a steady-state approach to wrangling dotfiles.
To expand on this, when a program gets installed, it should write its binary location in a file, then if you ever erase it, a doesthisexistanymore $Config/dir/*/symlink | remove_entire_configuration_directory should be run via cron daily.
No manual cleanup, ever.
xdg-user-dirs-update --set MUSIC ~/.xdg-user-dir/musicNope, fuck that.
This is my $home, I'll make it look how I want.
However, creating these folders by default is actually beneficial to most users. Most users (especially the vast majority that transitioned from Windows) expect these folders to exist.
I believe the choice to not create these existed and still exists. Yet either all the distros that made that choice are history or they all made the opposite choice.
I can say the same about other choices. I vastly prefer having a TRRS headphone jack and expandable storage on a phone. These do not appear to be priorities for most people.
It is true that applications will create these directories if they don't exist. But you can fix this by setting the XDG user directories[0] to something that isn't in your home directory. I set them all to a subdirectory under `~/.xdg-user-dir/`, but you could just as easily set them to `/tmp`. Nothing has recreated ~/Documents on my system in years.
I have long since accepted that it’s not my $home after all, but rather a general purpose application config dumping ground. My actual “home” is now a top level directory used only by me.
These presets only serve to confuse or at best annoy. These directories were well-intentioned originally but proved to be a huge mistake. They need to go away.
(Edit: The above rant applies only to Downloads/Music/Video/etc. I think ~/.config and ~/.local are fine. I have no problem using those as intended, and I've seen plenty of other people using them too.)
It is a little weird to see htem on a server OS install though
Apple software tries to organize files but it's a disaster: iTunes used to download movies to ~/Music/iTunes/iTunes Media/Movies, now TV.app downloads them to ~/Movies/TV/Media/Movies (!), and videos I shoot on my iPhone end up in ~/Pictures.
Should my game saves go in Documents? What about Documents/My Games/Publisher? What about %APPDATA%/Publisher? Hell there's even a few in ~/My Games. Every new game, figuring out where the save data went is an adventure and it's cluttered up every folder it touches. And now for whatever reason everyone is being pushed to put user data in %APPDATA%, which I don't consistently remember how to get to without that shortcut. So who knows!
I just looked up the save data location for Subnautica and it is (I shit you not): %AppData%\..\LocalLow\Unknown Worlds\Subnautica\Subnautica\SavedGames
Edit: clarified what I mean by the Appdata folder
IIRC at one point you couldn't even open one of the "AppData" homefolder aliases, Explorer would spit out some error message and you'd have to navigate to the hidden folder or key it in manually.
Really, you use all of them? When you go on vacation and take some pictures and video of your family, do you then split those files between ~/Video/Beach2022/ and ~/Pictures/Beach2022?
I've never seen anybody do this, not on Windows, MacOS, or any linux DE. Instead you get something like ~/Desktop/Beach2022 or ~/Downloads/vacations/Beach2022 or generally, a single directory anywhere (often on a USB harddrive, not their home directory at all), but never do those vacation files get split by filetype into ~/Videos and ~/Pictures, that seems like utter madness to me.
Not a safe assumption, annoyingly. Debian doesn't use tmpfs for /tmp; I guess they expect you to use /dev/shm if you want a tmpfs.
(Well, it exists in freebsd, but not openbsd, no clue if it exists in netbsd/minix)
~/Documents is there because Freedesktop.org somehow thought they were dealing with Windows and tried to port the solution of problems Linux does not have. The other directories aren't used on any OS and only serve to annoy people.
What do you do instead?
I organize my files by subject, not file type.
My files about space stuff go in a directory called 'space'. Files about the space shuttle go in space/shuttle. This directory contains any sort of file related to the space shuttle, be it picture, video, pdf, epub, or otherwise. I don't see any utility in breaking this directory apart into ~/Pictures/shuttle, ~/Videos/shuttle, ~/Documents/shuttle... Do you really do something like that? How far do you take this "different files, different hierarchies" principle? Do pdfs and docx go in two different hierarchies too? Are pngs and jpegs separated? What's your criteria for deciding when different sorts of files get different hierarchies?
Likewise, I'm not using ~/Documents just for stuff with an .odt or .pdf extension. Rather, any sort of human-readable text based file goes there. I have some markdown files that have embedded images, and of course the .png or .svg files exist alongside them in the relevant directories.
~/Music is probably the best example of why the default XDG approach makes sense. You can place everything in an Artist/Album hierarchy, and most music software is written to automatically pick up and scan these directories to keep their internal libraries in sync. There's no fuss.
I don't limit myself to just the XDG directories when there's not a close fit; i also have a ~/Projects for various things I've made and ~/Programs for statically built programs that can't be installed, and so on.
Basically, I've found that the XDG approach does a pretty good job of splitting up files by purpose, allowing you to compartmentalize by avoiding having too many directories at the top level of the hierarchy.
The concept that files are cross-compatible, and usable outside the app that created them, is increasingly foreign to an online app oriented world.
I am more likely to do something with those files than if they are hidden away in Downloads (which is like a tempdir), or on windows, some User/Documents subfolder somewhere.
I wish I could! But ~/Desktop and ~/Downloads keep getting re-created by ... something :-/ I tried setting XDG_{DESKTOP,DOWNLOAD}_DIR but it seems to get ignored. Been annoying me for years.
Perhaps something like:
ln -s /dev/null ~/Desktop
ln -s /dev/null ~/Downloads
etc.
Overall I think it would be productive to go back and redesign these old folders, even if it breaks some stuff if only to better reflect the way the new generation of consumers understands tech. There are probably a lot of young people who have never used a PC before, and will be confused coming into them with a background of smartphones and tablets.
Maybe we should eliminate the Desktop folder completely, and only allow adding widgets/app launchers/shortcuts to the desktop rather than files. Using the desktop as a folder has always been a questionable practice since you can't even see it when you have a window open.
At the very least, we should get rid of the "Templates" folder. I still have no idea wtf that's meant to be used for.
Seconded. I'm irrationally annoyed that it gets created and sometimes filled with .desktop files while my DE doesn't even use it.
> At the very least, we should get rid of the "Templates" folder. I still have no idea wtf that's meant to be used for.
You can place files in the "Templates" dir to serve as templates for the "right-click > new document" feature in nautilus and some other file explorers.
https://github.com/peteryates/dotfiles/blob/master/xdg/.conf...
Using unveil() and pledge(), OpenBSD hides many $HOME and system directories from Firefox and chrome. Thus an errant javascript will only see things in ~/Downloads
So one nice use for these.
One soundtrack I really liked was .hack, including the leading dot. Given that "hack" part and the overall theme of the games I assume the initial dot was quite intended. But what the creators could not have foreseen was that their soundtrack was missing from the majority of psf mirrors (but some had it obviously). I guess some part of some stack ignored dot-prefixed files when doing the mirroring.
Anyway, although I'm usually opposed to inband signalling, I am a fan of dot-prefix for hiding. Filenames already have so many special corner cases that you would have to treat them very carefully even without this one. The fact that it is embedded in the name means that any system can roundtrip hiddenness - something that cannot be said for metadata.
$ ls -R1 local/etc
local/etc:
bash_aliases
bashrc
gitconfig
inputrc
nanorc
ocamlinit
ssh
unison
XCompose
local/etc/ssh:
config
local/etc/unison:
backup.prfUnix design was about making things work (1) and simple (2).
http://files.catwell.info/misc/mirror/the-unix-programming-e...
I’m on the verge of abandoning the home directory. I’ll put My files elsewhere and leave home to the apps.
I thought XDG would surely solve this problem but it’s painfully clear now that most developers just don’t care. `node_modules` and `snap` are good cases in point. They didn’t even bother hiding them! LOL
I recently managed to crash Visual Studio 2022 because I had a symlink to a drive with a hidden $RECYCLE_BIN on it.
If you copy files in Windows the system will copy the contents of symlinks, however many backup programs will not follow symlinks.
So it's just an assumption and so the title is misleading
Also, dotfiles don't work well with masks because `*` doesn't match dot-files, and `.*` match all dotfiles including . and .. which causes many programs to fall into endless recursion (for example, `du -sh .*`).
Everything related to dotfiles in Linux is done wrong.
mv "$HOME/.config" "$HOME/config"
ln -sfn "$HOME/config" "$HOME/.config"
mv "$HOME/.foo" "$HOME/config/foo"
ln -sfn "$HOME/config/foo" "$HOME/.foo"
export XDG_CONFIG_HOME="$HOME/config"
https://specifications.freedesktop.org/basedir-spec/basedir-...
To me, it's an advantage to surface the latent problems, so I can message the authors to ask them about updating. So far, all authors are good with updating.
It turns out the bulk of the issues are in internal scripts that hardcoded ~/.config instead of using XDG_CONFIG_HOME. This is an easy fix, and at the same time we shellcheck those scripts.
But any consumer-facing OS will need some sort of mechanism of hiding/protecting configuration files/logs/etc. from deletion by clueless users, like how macOS has started hiding ~/Library by default.
But I agree with Pike. ls -a in the home sir, becomes an unorganized mess with all those apps that don't use the xdg default .config folder.
So I look at that as a reframing: do I have a good rebuttal? No. I have my own muscle memory and comfort factor, but that's all. How would I think if dotfile hiding didn't happen?
Hiding . & .. is arguably as silly. It's not like they can ever not exist but why hide them?
If not in /etc, then at least /usr/etc? I'm sure there is some historical reason but some brief searching and I've not really come up with why dotfiles everywhere are preferable to a centralized configuration directory.
If we used only /etc, then we would need per user entries there, which would be far more difficult, esp. with network-mounted home folders, especially when those are automounted.
The ~/.* convention worked a long time ago, far before either union mounts or namespaces were reliable.
(On a somewhat related note, I was very excited that when HP bought Apollo they might integrate the // resource naming into HP-UX. That would have been nice. I say related, because there is a great deal of cool from other systems that died with those systems.)
How bizarre. I had no idea it was actually related to the . and .. files.
An old way, but also it's still the way to hide a file, on *nix systems.
(Hence the "mistake", and the countless hours mentioned in TFA are still adding up, as people forget them when they shouldn't, or don't exclude them when they should…)
But it all makes sense, Windows is CP/M based, not Unix based.
a few decades ago macOS, amiga and atari were rather common, which all don't follow that convention. i don't know about BeOS/haiku. but if that hides dot-files then it's because they ported a lot of unix tools for commandline use.
i could not find anything about OS/2 but i suspect that it worked like windows.
I can't see how to do a git add to add all new files, including dot files, except those in .gitignore.
For the curious (was a bit annoying to find since gplus is down now and all links redirect to a different subdomain):
And hidden files seem to be an important thing. Some things you need to hide from the end user.
On the other hand, dotfiles convention is useful for meta/"parallel" data, `.git` being prime example - something tighly linked with abitrary file structure.
It would be so much cleaner if the convention would've bin similar to /var, /etc, and so on. ~/.var/cache or ~/.etc/
* im not joking, this is what un*x culture appears to essentially boil down to. boomers were all hippies in the 70s despite presenting themselves as "austere" and "mature" "wise" individuals now, which had the unfortunate side effect of too much drugs. further research is needed to clarify whether brain damage is caused by drugs or un*x
any encoding in file names at all is a mistake. this includes character encodings too, which may sound like a non-sequitur but its not. in fact, files should only have unique identities, not names. names should be metadata. folder hierarchy should just be a user defined data structure that can reference these said "files". anyone who tries developing their own OS without copying extremely idiosyncratic un*x ideas and without being bogged down by the overhead of assembly language or C quickly learns this. an example of youngins discovering this is IPFS (and subsequently implementing it on top of broken un*x)
You didn't get a job at freaking AT&T Bell Labs by being a pot smoking hippie. They were the squarest of square professionals--suit, tie, the whole bit.
alias ls='ls -A'
I tend to always start with:
ls -lArtWhat a strange fingerprint I imagine all our collective accumulated behaviors project about our past experiences.
I think this overall situation is most acutely a lack of standard. Someone up in the nosebleed section rightly pointed out that there should be some environment variable for a config to be written to, and I agree. I also want to stress that there should be the inclusion of a single variable on top of that, that all programs should use when constructing the path to their configuration. USER_CONFIG_DIR or something equally arguable should, if unset, default to $HOME, and if set, be used as the root to construct whatever structure therein is desired, also able to be changed with an environment variable specific to that program. Be it filename or further subdirectory, first using a single reference that is /not overloaded/, like HOME, but defaults to HOME if unset.
Secondly, I think we have all been there. We introduce a bug in an interface/API but then the users of it start depending on this behaviour and before you know it it’s a feature and it’s impossible to fix without breaking the world.
Re: Go “The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.” -- Rob Pike
I agree it's really a cringe way of approaching people. In this (dotfile) case there may have been some self-deprecation in the comment though, so it's frankly hard to get too miffed about it.
In my experience at AWS people that graduated didn't understand the languages we used. This leads to a bunch of problems, and makes it's easy to write very bad, unmaintainable code. Lots of needless class hierarchies in Java, or a misunderstanding of `this` in JavaScript. Go solves the problem perfectly.
With new grads, I generally didn't have to teach coding knowledge. It was more about transmitting wisdom around best practices, estimation, architecture, etc.
If anything, to me Go makes a lot of things harder that could be simpler. Lack of generics means boilerplate. Eschewing test frameworks means boilerplate. The structural typing stuff is exotic (but neat I guess) and not familiar to most new programmers. The actor / CSP type concurrency stuff is also exotic and not familiar to most new programmers. I don't see how it's a language for new programmers.
It also seems very much like a rehash of the concepts they had in Limbo so I actually don't find it honest for him to be claiming it was designed for this purpose (to simplify programming). Instead it feels very much like they had a hammer in search of a nail, an itch they've been scratching since the Plan9 days. And that's fine. I just don't buy this pitch about its market niche and the stuff about the relative intelligence of programmers is gratuitous. I've seen some very smart new grads produce amazing Rust code for example.
Anyway, the idea isn't that everyone already knows Go because it uses familiar features for concurrency, typing, etc.
The idea is that Go is such a simple language, that anyone with solid CS experience can pick it up in a short amount of time and start being productive with it.