Dotfile madness (2019) | Hacker News Reader

Dotfile madness (2019)

0x46.net

192 pointsmetadat181 comments

> You will most likely want to create a default configuration file with sane and sensible default values the first time your program is executed.

Please don't. Include the example configuration in your docs (/usr/share/PROG..) and mention it in the manual. Statistically user will have hundreds of apps installed but will only ever customize a small fraction them. So creating config unconditionally on first run is one of the worst contributors to homedir pollution.

Another argument is version upgrades: if you create configur on first run, you forever bake in the set of options present in that version. If later a new option is added, user will have no idea about it.

Also, idempotency of a tool is a desired property that tool devs often invest a lot of effort in maintaining. Initializing default config on the first run is a trivial way to break it.

It's much better to just have a sane implicit default value for each setting when config is not present. Then you can list the default config in your docs or offer a CLI flag --print-default-config or --copy-default-config.

Creating a file if it doesn't already exist is idempotent.

Only if you always create the file with the same contents, which is almost certainly not going to be true when the software evolves and adds new options.

I agree, at least if there is nothing that absolutely has to be specified for the program to work.

I don't see much of a problem with them being initialized in $XDG_CONFIG_HOME/<program>

You upgrade the software and it no longer works right. No one else reports the same issue. Turns out the version you tun for the very first time wrote a config in a location you never looked, and that config is incompatible with latest version.

What I personally find more egregious are dotfiles and dot-dirs on Windows, created by half-baked ports of Unix-first software (such as `ssh`).

Windows has `%APPDATA%` and `%LOCALAPPDATA%` that are accessible with `ShGetKnownFolderPath`[0], using the appropriate `KNOWNFOLDERID`[1]; all programs should put their configuration and cache data in these paths.

Windows also doesn't know what the dot at the beginning of a file/folder means. Instead, it and NTFS have something that's arguably better: a hidden file attribute `FILE_ATTRIBUTE_HIDDEN`[2], that's again settable/gettable when a file/directory is created or polled, rather than regex-ing for a full-stop.

The worst irony is Microsoft software creating dot-files and dot-dirs on Windows[3][4][5].

[0]: https://learn.microsoft.com/en-us/windows/win32/api/shlobj_c...

[1]: https://learn.microsoft.com/en-us/windows/win32/shell/knownf...

[2]: https://learn.microsoft.com/en-gb/windows/win32/api/fileapi/...

[3]: https://github.com/dotnet/sdk/issues/8678

[4]: https://github.com/OmniSharp/omnisharp-roslyn/issues/953

[5]: https://github.com/microsoft/vscode/issues/3884

Is “..” the official way to get a parent directory on Windows? Or do they have some other API and the “..” is just a nod to UNIX compatibility?

".." works fine in Windows environments.

> on Windows

There are many ways to access the filesystem on Windows. CMD.exe and PowerShell respect `..` to go to the parent directory, as does MSVC C++17 with `std::filesystem`, and .NET, with `System.IO.Path.Combine(String, String)`.

In PowerShell and .NET, there's also the `System.IO.DirectoryInfo.Parent` property, and of course, the equivalent `std::filesystem::path::parent_path` in MSVC C++.

There's an important consideration 'ChrisSD mentions here[0], i.e. whether resolving ".." is done lexically or by querying the filesystem. You may get different answers in each case if e.g. you try to resolve /foo/bar/../baz, when /foo/bar is a symlink to a different directory.

At least in case of std::filesystem, everything in std::filesystem::path is purely lexical. So, to get actual parent of some path, you'll need to go through std::filesystem::directory_entry and std::filesystem::read_symlink().

[0] - https://news.ycombinator.com/item?id=33312712

The difference between Windows ".." and unixy ".." is that the former is done lexically whereas the latter is a link to the canonical parent.

E.g. on Windows: "path\to\..\file" is always the equivalent of "path\file". Whereas on unixy systems "path/to/../file" may take you to a completely different parent directory.

That said, Unix shells can muddy the waters a bit. They may choose to act lexically with some commands.

> E.g. on Windows: "path\to\..\file" is always the equivalent of "path\file". Whereas on unixy systems "path/to/../file" may take you to a completely different parent directory.

Worth noting here that Plan 9 is using lexical names[1], and the Go standard library offers the function path.Clean[2] to achieve similar results. If your language/library/OS doesn't offer this behavior by default, it's likely a good idea to add/reimplement this function, unless you need bug-for-bug compatibility.

[1]: https://web.archive.org/web/20220824070659/https://9p.io/sys...

[2]: https://golang.bg/pkg/path/#Clean

It seems the only official native API for getting the parent directory from a known (full) path is PathCchRemoveFileSpec. But certainly C# and other similar platforms have functions doing similar things, and usually don't require a system API call anyway, it's just string manipulation - unless of course the path is relative to the current directory.

Hold your giggles until you learn that windows MAX_PATH is only 260 characters. Not just filename, the entire path.

Unless an app uses \\?\ prefixes and special API for navigation, which is rare, ime.

When using .Net 6 I use

  Environment.GetFolderPath( SpecialFolder.LocalApplicationData)

in Windows this maps to

  ~/AppData/Local

An in Linux to

  ~/.local/share

so it is configured well

I'm not sure I'd call that configured well. On Linux, share is supposed to contain static data used by the app, not the changing configuration that normally goes into appdata on Windows. For config the equivalent should be ~/.config/appname

He's getting the wrong folder on Windows too because that's what he's requesting when passing in SpecialFolder.LocalApplicationData. SpecialFolder.ApplicationData returns %USERPROFILE%\AppData\Roaming on Windows and ~/.config on Linux.

On Windows anyway, the root of the user profile is already filled with garbage directories, and so a few more dot-dirs and files feel par for the course. The advantage is that accessing your root dir is easier than typing out %appdata% in File Explorer.

One could say the same for ~ on Linux. 'It's already bad' isn't an excuse for sloppy programming.

Software should ideally adhere to OS norms. Better still, software shouldn't be opaque about where global/user-specific cache/config data is stored, and should prompt users to choose (or set sane defaults, again adhering to OS norms).

IMO the video game industry is a big offender of home-folder pollution, with saves and configs being splattered all over %USERPROFILE%\Documents, %USERPROFILE%\My Documents\ (which is a legacy holdover from Windows XP), %USERPROFILE%\Saved Games (which does have a corresponding KNOWNFOLDERID), etc etc.

If it's any consolation, games from Steam on Linux leak files all over the place on that OS too.

Right now the offenders in my $HOME not following $XDG are games, LibreWolf via Mozilla, Xmonad, GnuPG, and anything related to npm (with their maintainers closing any issued opened about support $XDG).

SteamApps/userdata, %APPDATA%/Publisher, %APPDATA/Developer, %LOCALAPPDATA%, the list goes on...

When making games, I have them store their config in %APPDATA% on Windows, although I do see the downside if there's a bug and I want to direct a customer to send me a log file that was saved there or just generally delete or move files around in there. It's not as familiar and safe-feeling for a nontechnical user to navigate to %APPDATA% is it is to just look in their Documents.

I prefer the app placing the config in the same place on every system. It makes it easier for me to find the file regardless of OS (to modify or back up with a script) and it's simpler for the app dev to write the code for as well.

I recognize this is probably because I first learned about config files on a Unixy system, it's not the "native way" for Windows, and other people have stronger cases for their preference, but this is my preference and I'm stickin to it :)

Your argument breaks down even between 'Unixy' systems. Linux programs that follow the XDG base directory specification write to those, whereas macOS programs tend to write to ~/Library/Preferences or ~/Library/Application Support, neither of which exist on Linux, or even BSDs.

When in Rome, do as the Romans do.

I’m a Roman and have no clue what these AppData/Romaning are for. Dotfiles that I’m aware of are my files and I expect them to reside in my ~ atrium.

> When in Rome, do as the Romans do.

Yeah, but then again, when you visit Rome for a short vacation as a tourist, the English versions of menus and such help, no?

Just saying, some degree of consideration for non-natives is a good thing, as it will help the tourism business generate more revenue. Same for OSes: give me at least an option of staying with my old habits until I get used to the new ones, then there's a higher chance I'll switch.

> I prefer the app placing the config in the same place on every system.

That is impossible because in Linux there is no drive letters and you cannot have the same path to a file as in Windows.

I don't find it egregious at all. Nobody should go out of their way to integrate with Windows APIs when files work just fine everywhere.

It’s basically the same as looking up $XDG_CONFIG_HOME on GNU/Linux.

That's just an environment variable while this ShGetKnownFolderPath thing is a Win32 API function. People should avoid directly depending on that stuff as much as possible.

People aren't obligated to "comply" with the XDG specification either. It's not a bug to just use $HOME.

You're directly depending on platform-specific stuff the moment you look up the user directory, because it's not $HOME on Windows either.

There is a difference between being cross-platform, and being Linux but mangling it so it technically works on other platforms.

It's just a different variable: USERPROFILE.

https://stackoverflow.com/a/42700844/512904

> being Linux but mangling it so it technically works on other platforms

Sounds like the easiest way to solve this problem. I don't blame or shame developers for choosing it.

This is simple and works everywhere:

  char * home = getenv("USERPROFILE");
  if (home) {
      // Use home directory
  }

The answer demonstrating the Windows API way has the following code to accomplish the same thing:

  WCHAR profilePath[MAX_PATH];
  HRESULT result = SHGetFolderPathW(NULL, CSIDL_PROFILE, NULL, 0, profilePath);
  if (SUCCEEDED(result)) {
      // Use home directory
  }

Yeah, I seriously doubt open source developers want to deal with this Windows bullshit. I sure don't. Maybe if you're getting paid.

'This Windows bullshit' is merely the Windows equivalent of a veritable zoo[0] of Unix/POSIX/Linux headers and functions including unistd.h, sys/types.h, sys/socket.h, fcntl.h, sys/ipc.h, semaphore.h, stdarg.h, signal.h, pthread.h, and so on.

Just because you aren't used to it doesn't mean it's 'bullshit'. Might I provide a counterexample to the 'simplicity' of Linux? Developing video games natively on Linux is such a pain point that the industry has standardised around simply translating Windows syscalls and Direct3D shaders to their Linux equivalents, because the latter is so heavily fragmented to develop natively for. Also known as Proton.

[0]: https://en.wikipedia.org/wiki/C_POSIX_library

Whenever dotfiles are mentioned, it's safe to assume we're talking about terminal applications and that the target audience is software developers. This stuff only really matters to us, people who actually care enough about the system itself to not only use a terminal but also to bikeshed configuration formats and file system trees. That's exactly what the article this thread is about and the comment I replied to were talking about: stuff like developer tools and ssh.

Honestly I don't even disagree with you. I think POSIX sucks. I think even the libc sucks. However there is no way you can look at the Windows API mess above and think it's somehow better than the portable and clean getenv function call. Maybe you can "get used" to that cruft but I can't. If I had to deal with code like that the first thing I'd do is abstract it away so I don't have to deal with it anymore. That's exactly how Windows is and should be treated by everyone.

Video games, your counter example, are probably the most platform-specific software imaginable. There's a reason pretty much everybody uses an engine these days. There is no way you can compare that stuff to getting a user's home directory and writing files in there.

Also, for video games it ultimately doesn't matter how sucky the APIs are since it's all about market share anyway and people will use whatever they need to get it done. First parties don't put in effort into porting games to Linux because they don't see the money. Valve puts in effort because it gives them their own platform that they control and therefore leverage over Microsoft and its app store. Emulation and compatibility layers are a proven solution when first parties aren't playing ball. If they don't want to make it work on Linux, people will make it work whether they want it or not.

The former does not in fact work everywhere, it only works on Windows. Once you have your #ifdef, there is absolutely no point in having the wrong Windows-specific code instead of the right Windows-specific code. Also, look at what you're describing as weird bullshit: a function call with an enum, instead of a function call with a string. The only difference is that you haven't bothered to familiarize yourself with the world outside of Linux, so it feels unfamiliar.

Also, you're comparing something it makes zero sense to compare. People are not saying 'you need to fetch $USERPROFILE without calling getenv'. People are saying 'you need to not store random garbage in $USERPROFILE'. Calling getenv("APPDATA") is a perfectly fine way of fetching FOLDERID_RoamingAppData, the same way you weren't going to bother calling getpwuid if $HOME was undefined.

> The former does not in fact work everywhere, it only works on Windows.

It's just some example code. Real code will probably try a number of variables.

> Calling getenv("APPDATA") is a perfectly fine way of fetching FOLDERID_RoamingAppData

Okay then. Top level post said you needed a Win32 function for this. If you can do that without those functions, I suppose it's easy enough to support.

> calling getpwuid if $HOME was undefined

$HOME is set by login and ssh to whatever is in the passwd database. Can it ever be undefined? Seems like something is seriously wrong if it is.

> Nobody should go out of their way to integrate with Windows APIs

  int writeConfigFile(ConfigFile file) {
  #if defined(_WIN32)
      // Windows-specific function
  #elif defined(__linux__)
      // Linux-specific function
  #elif defined(__APPLE__)
      // macOS-specific function

With modern C++, there's probably a way to do this in a nicer manner, and achieve runtime polymorphism with virtual functions or a lambda (function pointer) table.

As another sibling commenter wrote, .NET already has a platform-agnostic function to write to a `user program data` folder that nicely maps to the platform-specific directories. Other GC languages (Java, Go, etc) likely have similar abstractions.

As for scripting... A cross-platform project with scripts that also targets Windows should (in my opinion) have PowerShell `.ps1` scripts accompanying any `.sh` scripts. Better still, scripting should be done in a language that's truly platform-agnostic, like Python.

> With modern C++, there's probably a way to do this in a nicer manner, and achieve runtime polymorphism with virtual functions or a lambda (function pointer) table.

You proved my point. Creating an #ifdef mess with polymorphism, virtual methods and indirection is the very definition of going out of one's way to support something that really doesn't need any support.

If a developer writing in C/C++ is targeting Windows, then bringing in `<windows.h>` (and associated headers) is almost guaranteed, especially if said developer wants to work with native APIs (unless said program will rely on MinGW/Cygwin/Msys2, which are arguably worse, since they drag in an entire Linux directory tree per application).

This is just part of the cost of developing cross-platform, and I'm sorry, but to me, dismissing this as 'going out of one's way' is just being sloppy.

Are the developers of "Unix-first software (such as `ssh`)" targeting Windows though? I don't think so. They don't deserve to be shamed for this stuff. They aren't "sloppy", Windows is lucky to get ports at all.

> #elif defined(__linux__)

I felt a great disturbance in the source, as if hundreds of BSD users suddenly cried out in terror and were suddenly silenced.

No need for runtime polymorphism for this. The same binary won't be used in different operating systems. This is a classic example of misuse of C++ features just because they exist and a reason why I personally prefer programming in C. What I would do in a case like this is to have three source files each implementing the function for UNIX/Windows/Mac and the build system should take care of compiling the right one into the program depending on the OS.

This is a much better idea, actually. You're right that what I mentioned is a bad misuse of C++ features.

I initially did think of simply letting the build system handle it (CMake can do this excellently, as can Meson, probably) by simply choosing different source files for different target OSs, but when typing out the parent comment, I forgot about it in my train of thought.

Come to think of it, this is far more elegant.

The reason app developers think it's fine to dump them in $HOME is because Linux hides them. Those app developers do not bother hiding them on Windows. The specific functionality being used does not 'work just fine everywhere'. It is not the case that Linux functionality is How Computing Works with every deviation being 'out of the way', not even in a hypothetical world where Linux isn't the drastic minority of machines by an order of magnitude.

> Linux hides them

Linux doesn't hide anything. It's POSIX and GNU programs that choose not show anything that starts with a dot by default. As far as Linux is concerned they're just normal files on the file system.

It doesn't have to be that way. Personally I have my system configured so that all "hidden" files are shown because I hate the concept of hidden anything. I used to do this in Windows too.

> The specific functionality being used does not 'work just fine everywhere'.

It kind of does. "Program reads configuration from a file" seems like a really basic thing that every operating system supports. There really is no need to directly depend on any Windows API stuff. Even worse would be suggesting the use of the registry.

> It is not the case that Linux functionality is How Computing Works with every deviation being 'out of the way', not even in a hypothetical world where Linux isn't the drastic minority of machines by an order of magnitude.

And yet Windows is the system getting "half-baked ports of Unix-first software" here. Why should the developers be shamed for not going of their way to support Windows conventions for Unix software?

People develop software on and for the systems they actually care about. Windows is the legacy proprietary system people only support because it gets them paid. At least I hope someone got paid to port ssh to Windows because I sure wouldn't want to do it otherwise.

> Even worse would be suggesting the use of the registry.

I would actually argue otherwise: the registry is the perfect place to put configuration data, provided Microsoft's guidelines[0] on registry key creation/deletion and hive use are followed. I would argue every OS should have its own 'registry' for a universal location to set configuration for programs.

On the other hand, if every application has its own configuration file, then every application shall have its own configuration file parsing/serialising/deserialising routine. It's also what has led to this explosion of 'config file formats': .json, .yaml, .ini, .conf, .cfg, .toml, .xml.

I guess this is why people dislike systemd so much: it reminds them too much of Windows.

[0]: https://learn.microsoft.com/en-us/windows/win32/sysinfo/regi...

You appear to have an attitude of UNIX/Linux > Windows, which is unfortunate. The two OS families are different, but I personally believe neither is superior to the other. Both OSs suck in their own ways, both OSs are better in their own ways (this conclusion after using Windows and Linux both on a 70:30 time ratio).

Windows is in every way as modern an OS as Linux is, but just like Linux, has inherited strange baggage from its 40-year-old past, like 'a file cannot have COM in its name'.

> the registry is the perfect place to put configuration data, provided Microsoft's guidelines[0] on registry key creation/deletion and hive use are followed

Few people follow that stuff. I wouldn't be surprised if Microsoft's own people didn't follow that stuff. The result is of course a mess that users have to clean up with software like CCleaner or whatever people use these days.

It sucks even if those guidelines are followed. Configuration is data and belongs to the users. I know people who really like portable applications that they can keep on a USB drive and use anywhere and this registry bullshit breaks that.

> I would argue every OS should have its own 'registry' for a universal location to set configuration for programs.

And I would argue otherwise. The Windows registry is essentially a poor excuse for a proper file system. I think at least one Linux desktop environment tried to reinvent this crap and it sucked just as hard.

> every application shall have its own configuration file parsing/serialising/deserialising routine

The Windows registry values are essentially memory dumps of C types. It's perfectly possible to simply dump raw bytes into binary files on Linux too if you don't care about stuff like endianness. The result would be strictly superior to the Windows registry because the files are being stored in a real file system.

> I guess this is why people dislike systemd so much: it reminds them too much of Windows.

I like systemd.

> Windows is the legacy proprietary system people only support because it gets them paid.

Windows is the system everyone actually uses - even developers!. I write software because I care about people using it.

> Why should the developers be shamed for not going of their way to support Windows conventions for Unix software?

Because it's wrong on Unix, and even wrong on Linux, and once you are doing the right thing on Linux it is no longer out of your way.

Regular windows users do not visit $HOME often either. They work mostly in Desktop, Documents and Downloads. They can’t even “go up” from these anymore for more than a decade. None of the non-tech users I know would even understand me if I told them to look at their homedir. So yeah, it’s fine regardless.

Btw, many native, no-linux windows apps, including Microsoft ones love to create their bs folders like “My Games”, “Microsoft Visual Studio (c)(tm) <yearname> Pile of Whatever” or “Android Projects” right in ~/Documents. This “way” is unenforceable even in OS’es own company.

I think "hidden" files are a bad idea. What's the point of hiding some of the files in a directory from user? It only makes things more confusing.

For example the folder might look empty but in fact it could contain thousands of hidden files. You want to delete it as it is empty but accidentally delete important system files.

Even worse, imagine if all files on an USB drive are marked as hidden. You see that the drive is empty but when you attempt to copy something to it see a message that there is no free space.

One more example. Imagine if you have a project and want to edit an .env file. But as dotfiles are hidden in Linux you don't see this file and cannot open it. It turns out that Windows Explorer which doesn't hide dotfiles is much better for developing projects with Docker!

There should be no hidden files. We cannot do anything with Windows and their MS-DOS legacy but Linux could stop hiding dotfiles in GUI and CLI commands.

i just always list dotfiles for years now.. and my IDEs too. mostly a non issue.

And I have to enable showing them on every system I work with (and add -a to every ls command, and google which option enables incldues them in pattern expansion). That is annoying.

Well.. ya. I set up my aliases and then forget about it again. Globbing is mildly annoying but i think there's a dotglob option or something you can set

Hidden files on MacOS were used sparingly—the main instance of a hidden file is the file which stores an icon for a folder, if the folder has a custom icon. Hidden files are only hidden from the UI, they appear normally in the command line.

The standard Unix folders (usr, var, etc) are also hidden—to be honest, this is a compromise because the Unix filesystem hierarchy sucks, but we are saddled with it for backwards compatibility reasons.

I did appreciate the relative clarity of old macOS systems—everything in the OS was in “System Folder”, the rest of the filesystem was yours, and you could freely move applications without breaking anything, uninstall them by putting them in the trash, etc.

Hiding thumbnails cache is bad because a user might believe that the USB drive is empty and doesn't have any confidential data but the thumbnails cache might still have them.

There is no need to hide such files. Just display them as system files in Explorer.

> The standard Unix folders (usr, var, etc) are also hidden

Again there is no need to do it. Just move them to a directory named "System files".

You can’t move “var” to a “System files” folder. This would break programs which expect /var to be, well, at /var. Long ago on macOS, everything was in “System Folder”, and it worked fine, but macOS has since inherited a massive Unix legacy.

Thumbnail caches are not stored on USB drives, they’re stored somewhere else, away from your files.

For people out there on macOS, you can press Cmd-Shift-dot in Finder to toggle displaying any hidden files (including the standard UNIX hierarchy at /).

By the way, I'm pondering setting my XDG_*_HOME vars on Linux/BSD to use the macOS paths like ~/Library, just to see if the standard is actually being followed.

I did a similar thing. I set XDG_CONFIG_DIR=~/ (normally ~/.config/). It works surprisingly well, though I encountered some edge cases that eventually made me give up. Also some programs don't know about XDG_CONFIG_DIR and just hardcode ~/.config/, but you can symlink ~/.config/ -> ~/.

>What's the point of hiding some of the files in a directory from user? It only makes things more confusing.

If a user doesn't know what .ssh/ or .bash_history are, and it can hurt them to accidentally delete or modify them, why show them by default? It's like training wheels on a bike.

>For example the folder might look empty but in fact it could contain thousands of hidden files. You want to delete it as it is empty but accidentally delete important system files.

If it is possible for a regular user to accidentally brick a system like this, it was doomed to begin with. If a user doesn't know what a dotfile is they shouldn't have administrative privileges, and if they do get them there is no safeguard to stop them destroying the system. It's not a dotfile issue.

>Imagine if you have a project and want to edit an .env file. But as dotfiles are hidden in Linux you don't see this file and cannot open it. It turns out that Windows Explorer which doesn't hide dotfiles is much better for developing projects with Docker!

In what reality is someone developing software and building Docker containers who doesn't understand the concept of hidden files? Seems awfully contrived.

> If a user doesn't know what .ssh/ or .bash_history are, and it can hurt them to accidentally delete or modify them, why show them by default? It's like training wheels on a bike.

This is really the problem with the home directory. It’s supposed to be the place where the user puts their files, but it’s also a place where applications put their files, usually without asking the user.

The battle is lost - the home directory territory should be ceded to applications and the user should have a separate space in which to manage their files.

This can sort of be achieved with the $XDG_CONFIG_HOME environment variable, but not all software respects it, so in practical terms it’s a lot easier just to leave $HOME to be the application config dumping ground.

It kind of goes around to the same point: developers are sloppy and assume that ~ or %USERPROFILE% has full read-write permissions, and treat the corresponding home directories as a free-for-all dumping ground.

Yes! I did that long ago, putting my stuff in /home/kept/. I'm surprised at everyone who still puts their files in ~. What a mess.

> If a user doesn't know what .ssh/ or .bash_history are, and it can hurt them to accidentally delete or modify them, why show them by default?

Because they are there. To prevent accidental deletion you can show an icon with a skull and crossbones or name a directory "configuration files" so that it is obvious what's in there.

> if they do get them there is no safeguard to stop them destroying the system

If the folder are named or labeled as "operating system" then the user will probably understand that it is better not to mess with it.

> In what reality is someone developing software and building Docker containers who doesn't understand the concept of hidden files?

Hiding hidden files is annoying because you have to add -a to ls command, have to add an option in Bash to include then when matching against a star pattern and to configure Explorer so that it shows them. It would be easier without them.

> To prevent accidental deletion you can show an icon with a skull and crossbones

This will make users more likely to delete it thinking it's a virus.

> If the folder are named or labeled as "operating system" then the user will probably understand that it is better not to mess with it.

Big assumption that most people know what an operating system is.

> Hiding hidden files is annoying because you have to add -a to ls command

Have you ever heard of aliases?

You are suggesting a workaround instead of solving the original problem.

The workaround is trivial and painless, whereas “solving the original problem” would require fundamental changes to decades-old systems conventions. I was responding to the “annoying” part, which is effectively solved using an alias.

> If a user doesn't know what .ssh/ or .bash_history are, and it can hurt them to accidentally delete or modify them, why show them by default? It's like training wheels on a bike.

You have permissions to prevent from accidental deletions.

Personally, I think hidden files are OK, but we have lousy defaults, regardless of OS.

By default, Explorer hides hidden files (and goes one step further and doesn't display file extensions: why???), so do Finder, Dolphin, Nautilus, and even `ls` by default.

Well yeah.. because they are hidden files. They wouldn't be hidden files if the default wasn't to hide them...

hidden file extensions by default are to "save" users from renaming files and borking the extension. Later windows UX improved the renaming to help avoid selecting the extension.

file extensions are the mistake here really not them being hidden.

Executability based on extension combined with hidden extensions is imho one of Windows biggest mistakes. Make an exe, give it the icon of a word document - no way for a layperson to know it’s a trick.

"Show hidden files" and "show extensions for known file types" are two folder options I enable on any Windows system I touch / within a few minutes of installation. I'll take a bit of extra visual clutter if it means I can actually see what all is in a folder, vs. just assuming I know.

When I run ls I don't want to see all the configuration files. Just my files. I think that's the point of hidden files.

> One more example. Imagine if you have a project and want to edit an .env file. But as dotfiles are hidden in Linux you don't see this file and cannot open it.

How likely is it that someone is going to want to edit a .env file and not know how to view hidden files?

> When I run ls I don't want to see all the configuration files. Just my files.

Well then configuration files can simply not be in the same directory as your files. Easy fix.

Your user doesn't have permissions to write to any directory that isn't yours. It doesn't mean configs have to be in your "documents" directory, but they do have to be in with your files!

The comment I was responding to was talking about the utility of files being hidden.

But they don't need to be hidden by being special, they can just be "hidden" by living in a subdirectory that you don't look inside.

Re accidental deletion, ntfs also has a "system" flag. Windows prompts if you delete one of those.

Windows is a lost cause anyway. Even Microsoft products (at a glance: IISExpress, Visual Studio, RDP, Powershell) happily dump their cruft in the Documents folder. I don't know where they imagine users should actually save the files they control.

I completely agree! That's why I created this site: https://xdgbasedirectoryspecification.com. We need some place where people can immediately understand the practical benefit, and the minimum ask of their users.

Of course, it doesn't help just to complain about the issue in a void, so I've also submitted like 10 PRs and filed quite a few issues. Projects like pnpm and Poetry took note, and now they comply!

Someone's gotta clean up house, and I don't mind doing the dirty work.

The idea for your site is great. A quick suggestion: if you want everyone to immediately understand and sympathize with your cause, then use the tools that everyone knows in your examples. Using some long exa incantation risks alienating people that don't care about things like the latest trendy grep alternative. Just use: `ls -d .*` or `ls -a`

Thanks for the suggestion - agreed, I'll make that change

Using some long exa incantation risks alienating people that don't care about things like the latest trendy grep alternative

Exa is the latest, trendy ls alternative.

I don’t think you should remove the Exa example but you certainly should show how to use ls to do something similar for those people still living in the dark ages—ls was created in the 1970s and it hasn’t kept up with the times all that well [1].

I created an ‘ll’ alias for Exa using the Fish shell; obviously you make shell-appropriate aliases in Bash or Z shell:

    function ll
      exa --color always --long --no-user --no-permissions --all --icons $argv
    end

[1]: https://the.exa.website/introduction

I appreciate that you are trying to be helpful, but you might be over-evangelizing trendy things at the risk of appearing patronizing; exa is a thing like the latest trendy grep alternative, and the point of my original comment still stands.

No need to have a rosetta stone of alternatives to demonstrate an illustrative example where the basic tool that everybody already knows and understands at a glance works.

I'm not saying that exa, or fish, or ripgrep, or any other trendy tool is bad.

Why does the "fixed" example still list .profile? Why wouldn't the shell config live with everything else?

Mainly because it's easier to get people on board when we're keeping a few of the super core dotfiles, especially since those are kind of impossible to change - they are hard coded literally everywhere

Admittedly, the ~/.bashrc etc. files should be there too - but I wanted it to look cleaner :P

The following programs are for Linux only (The XDG Base Dir Spec isn't inherently followed for MacOS and Windows)

You should fix this. While macOS and its system applications and bundled applications don't adhere to the XDG Base Directory Specification; many programs that run on macOS do.

In fact, almost all of the programs you list run on macOS and (by definition) use the XDG Base Directory Specification. So I would clean that up to not give an incomplete statement.

BTW, the macOS directory structure [1] has been documented since macOS (MacOS X back in the day) shipped in 2001.

[1]: https://developer.apple.com/library/archive/documentation/Fi...

Hmm, that's why I said inherent? Because as you mentioned there is a separate macOS directory structure and the system utilities don't follow the specification. Also, the specification is only written for Linux in mind, which is why I said "not inherent for macOS.

Is there an exact phrasing you would prefer?

The short explanation is native GUI apps that use the macOS frameworks use the classic macOS directory structure.

The classic Unix utilities and apps typically use $HOME on macOS, while their modern, often Rust-based replacements—WezTerm, Exa, fd, broot, Fish, etc.—use XDG.

Vim uses ~/.vimrc while Neovim uses XDG.

I asked for an exact phrasing, not for an explanation, because I have already written exactly that under the "Should I do this on macOS?" heading...

Am I missing something, or should the Python example use "config_dir[0] != '/'"?

I.e. if it's empty, or if the first character is not a forward slash, then use the default?

no I think you're right, that was a typo

A lot of software that does not default to using XDG directories can be coerced into it, usually with environment variables or just manually creating config in a secondary supported XDG location - the arch wiki is the best source of info on that: https://wiki.archlinux.org/title/XDG_Base_Directory

Discussed at the time:

Dotfile madness - https://news.ycombinator.com/item?id=19063727 - Feb 2019 (514 comments)

I do not understand the problem with "dot"-files. And I do not understand the benefit to have them in another directory. I think there are more important problems to solve in this world.

Are there more important things to solve? Of course.

However, this particular topic on this particular message board is about dotfiles and the alternatives.

Dotfiles are a clumsy mechanism that came from the early days of Unix.

There are better alternatives that provide the same lack of "unknown/unnecessary" directories and files in the user's home directory/folder.

Each of Windows/MacOS/XDG-compliant OSs has the equivalent of:

1. A configuration directory 2. A storage directory that is maintained between user logins and/or application execution 3. A temporary/cache directory that is not guaranteed to be maintained between user logins or application execution.

The different languages/runtimes also have ways of retrieving these locations which are cross-platform but return the correct values and should be used by all applications that are written to modern standards.

This sounds like “let’s design something nobody ever will use”. Compare how often you (or anyone) created, edited, deleted or backed up a ~/.dotfile vs some file who-tf-knows-where in AppData or Library mazes. These have near zero discoverability and expect-ability.

There are better alternatives

Let’s define “better” first.

Surprised no one has mentioned nix home manager, which deals with dotfiles in an incredibly elegant way.

https://github.com/nix-community/home-manager

I've open the project link, scanned the docs, I could not figure out where do find a general description of the problem it tries to solve, and the solution advantages.

You made me curious and didn't deliver ! The anguish !

https://news.ycombinator.com/item?id=33313657

I did, but I shouldn't have because I was too lazy to explain what it is.

https://news.ycombinator.com/item?id=33311958

I think the real culprit here are operating systems and file systems, not programmers (of end-user applications). File systems were designed during an era where semantic meaning was just not important; all file systems cared about (and still care about) are directories and files. The whole paradigm should've been abandoned long ago for something much cleaner.

Like what?

I think the ship's long sailed. Too much software just throws random crap into $HOME, so it's best to give up. I've resigned myself to the state of affairs and my "real" $HOME is in $HOME/code $HOME/home.

As the saying goes, layers are only added, never removed. The world where people respect XDG is not one that can be achieved anymore.

Which software in particular? Given my experience, it is well within reach, besides a few exceptions including Firefox/Thunderbird, etc, and one relating to Chromium-based browsers.

The trick is to use the XDG Base Dir Spec, only if the old ~/.name location doesn't exist. Old users are happy because it doesn't break compatibility and new users are happy because nobody's dropping 10 sacks of potatoes on their metaphorical doorstep

Just a random sample from my home directory right now on a random dev server: .ssh, .gnupg, .vscode-server, .zshrc, .ipython, snap, .antigen, .pki, .profile, .python_history, and there's much more.

I don't know what's the situation with desktop linux, may it's better, maybe it's worse, but even on a headless dev server it's already a lost cause. Trying to convince everybody to move to a new standard is a hopeless task, imo, best to move on.

Oh right I forgot about those - I set some environment variables so I don't see any of those except for .pki

And in my opinion .ssh and .gnupg are okay since in some contexts, the "XDG variables" haven't been initialized in certain circumstances when using those utilities.

But yeah, I'll have to look into those - maybe there is something I can do to fix that

I think it is better to think about it as XDG failing to respect a world that existed long before it did. $HOME/.* namespace belongs to the user, but contents are special. It's almost like complaining about Linux enforcing the naming of certain file extended attributes.

what does "but contents are special" mean?

This is kind of a thing of the past with Flatpak, since apps can only create files within their sandbox's $HOME, which isn't the same as the user's $HOME. Now we just need deb/rpm/etc apps to be sandboxed in a similar way using bubblewrap (the tool Flatpak uses for sandboxing).

Yeah… but then you are using flatpaks.

If you'd have said this way back I'd have had my hands up like "What's wrong with flatpak", but as I moved to smaller hardware, I started to realize how much space a mainly flatpak environment takes up. It kind of makes sense in retrospect, so I'm not sure what I was expecting.

Eh, I dunno. XDG is just some group or whatever, dropping config files in .programname is the de-facto community standard. If people want to follow XDG that's fine, but I can't blame anybody who wants to ignore all that noise.

XDG has real practical benefits, though.

I personally hate the XDG directories. It’s annoying for different part of an application’s data to be split between `~/.config`, `~/.Cache`, etc. Rather than for all the data for the `foo` program to be in a file named something like `~/.foo*`

I find it useful; I might want to version-control .config but definitely not .cache. And if the program is broken I might blow away .cache but not .config.

That still works if it’s `~/.foo/cache`

    rm -r ~/.foo/cache ~/bar/cache ~/baz/data/app/cache [20 more apps..]

is less ergonomic than

    rm -r ~/.cache/\*

I think HN escaped the * in that final line before I edited it into a code block and my edit window is expired so I can't fix it

There’s always:

    rm -rf ~/.*/cache

Assumes the app uses cache as it's cache, rather than names like dlcontent, bananas_content, storage.sqlite, assets, meta, $TIMESTAMP.txt etc.

Well, I don’t mind standards, I just prefer all the files an app touches to be under a single root folder (that’s not just $HOME)

Asked out of ignorance, but do the envvars not allow one to easily co-locate data/config/cache by setting them all to the same value?

Not in a way that would be useful here, I think: I prefer ~/.firefox/cache to ~/.cache/firefox and this requires the variables being unique per program (although, you could probably wrap each program with a script that sets the relevant environment variables to achieve this effect)

I was thinking along the lines of ~/.data/{prog} without requiring further subdirs. The config, data, and cache would all coexist within the same directory per program

Each of the data, config, and cache envvars would all be set to ~/.data in this setup (though “data” isn’t a name I’d use)

It is convenient because you can delete ~/.cache any time without losing configuration files.

strongly disagree - as of 2022, more apps follow the XDG Base Directory Specification than not - in fact, by a wide margin. So by "doing dirty" in your users home directory, you're going against the grain of the ecosystem.

It doesn't seem like you understand much about the specification? I'd recommend reading it, or at least a short summary I've made available here: https://xdgbasedirectoryspecification.com/

My home directory has dotfiles for aspell, bash, bzr, cargo, Clojure, CPAN, CVS, Darcs, D-BUS, Dillo, DOSBox, DOSEmu, Emacs, Eclipse, Emacs, ESD, FLTK, fontconfig, FontForge, GForth, GHC, the GIMP, Git, GnuPG, Gnuplot, Guix, IceWM, Idle, Java, John the Ripper, Jython, Links2, MilkyTracker, Mozilla (Firefox), MPlayer, Nethack, Octave, Racket, R, Skeinforge, Slic3r, SSH, SQLite, Subversion, Units, Vim, W3M, WINE, X-Windows, XScreenSaver, and zsh, which is 51 "apps". Some of those have been there since last millennium, and it would not improve my user experience for those "apps" to look for my configuration file in two different places, or to stop looking in the place they've been looking for decades.

(God forbid I should use remote CVS again, but if I do, I sure don't want to have to debug where the fuck it's sticking .cvspass now.)

There are another 41 things in .config, including Bitcoin, Calibre, Chromium, DjVuLibre, Gomuks, htop, Inkscape, Mumble, OpenSCAD, Transmission, and other things. If I had to choose, I would say that those are the things "going against the grain of the ecosystem". But I think it's fine. As long as it doesn't change in another ten years to be .Config, or .conf, or Configuration, or either Configuration or Configuración depending on whether I installed the machine in English or Spanish. Because then I am seriously going to look into government surplus cruise missiles.

> it would not improve my user experience for those "apps" to look for my configuration file in two different places

Some of those already do, though. At least Vim and Emacs do, and I personally have zsh in .config as well.

Yeah, Emacs looking in both .emacs and .emacs.d has been a headache from time to time.

Isn't .emacs.d still in your $HOME? I thought the path was $XDG_CONFIG_HOME/emacs

Oh, is it doing that too, now? Ugh. Three possibilities is worse than two.

Yeah I sympathize with you - my goal is for the Linux community to use a system - that is, a well-defined one, not one born arbitrarily out of the (`if (name[0] == ".") file_hidden = true;` code that has since metastasized into an ungodly manifestation of chaos itself. And that good system is the XDG Base Directory Specification. I'm sure as you know, Unix systems in general have a bad history of ad-hoc protocols and conventions, and in the process of correcting for those, somethings going to be disrupted. I've done my part to ensure that a config location that exists will continue to be looked at, but as you mention, it's confusing and increases complexity. I'm sorry, but that is what I'm willing to risk to fix this particular underlying issue.

I do not share your fear of chaos or your faith in rational planning over ad-hoc protocols and conventions. I would prefer that you not attempt to rationalize the operating system I live in. Have you considered that the system you are looking for is perhaps not any kind of Unix, but maybe something like Plan 9, Macintosh System 7, PalmOS, or Oberon?

Instead of lamenting that Unix isn't more like Oberon, and trying to make it more like Oberon, consider that possibly the secret of Unix's success is that its ad-hoc protocols and conventions embody ideas that work very well, so instead of replacing them with untested rational planning, you should try to learn from them. Embrace the chaos.

"Everybody else does it" is not an argument.

If you read the parent comment, then you would know that I'm correcting the claim that not using XDG Base Dir Spec is the de-facto standard. It's not, so I'm simply stating that

XDG is like four env vars. Doesn’t seem like much noise.

> dropping config files in .programname is the de-facto community standard.

Fuck you so much.

There's no call for that. I don't have the ability to flag, @dang or others?

Posting like this will get you banned on HN. No more of this, please, regardless of how strongly you feel about config files.

Also, would you please stop posting unsubstantive/flamebait comments more generally? It's not what this site is for, and it destroys what it is for.

https://news.ycombinator.com/newsguidelines.html

If people want to follow XDG that's fine, but I can't blame anybody who wants to ignore all that noise.

Literally every program I’ve installed for the last several months--WezTerm, Neovim, bat, broot and many more just use XDG by default. Most of the time, the user doesn’t have to do anything, though they might initially be confused if they’re looking for their config files.

> XDG is just some group

> the de-facto community standard

Feels like you’re just selectively defining “community” to exclude XDG. Maybe there’s a good reason, but you haven’t really described it.

Whoever defined $HOME as a place to stick configuration files was also “just some group.”

Freedesktop (new name for XDG) is not the community. They are some people with a bad reputation when it comes to software quality and maintenance.

> XDG is just some group or whatever

So are ISO, W3C, IETF, etc.

> dropping config files in .programname is the de-facto community standard

Dropping litter and open-air defecation are de-facto community standards in some unfortunate parts of the world. That's not an argument to keep doing it.

Personally I like my dots but I am pretty novice and I do use GNU stow.

People are simply just lead by example: ~/.bashrc, ~/.bash_profile, ~/.profile, etc.

Indeed, the XDG specification is rather young in the history of UNIX.

The shell profile is a fundamental part of using a Linux computer. It affects how every single program is launched. This is not the same as a utility program's config file.

I don’t see how my prompt function or aliases are more fundamental than vim, git or other rc’s.

It affects how every single program is launched

That is true, but used as an argument for this subject is tangential and overblown. Look at your .bashrc and tell which parts are too fundamental to not be stored elsewhere, if really needed.

The solution is simple, stop messing your home and put personal data in

/home/you/personal/

source code in

/home/you/src/

The home is not a place supposed to store random "regular files"

On the contrary, that's exactly what it's supposed to be.

The $HOME in unix is a user data storage location

A lot of data is managed by software and by convection there is a dot before them

Unmanaged/Arbitrary data like random pictures or notes are personal and should end up in $HOME/personal

git repos are special managed data, thus they all end up in $HOME/sources

$HOME is supposed to let software to create their data folder and optimally without the dot

The data here includes config (which is read & write like data file)

$HOME/.config or XDG spec is monkey patch that do the wrong thing.

GUI file manager should open $HOME/personal as initial location

Terminal should open $HOME to let user access software's data file with shorter path (aka without .config/ )

You don't know if i am mad or being brilliant or simply brain dead, but i am making perfect scene.

Yeah but no putting the genie back in the bottle.

So the next best is to find another place as your home as the OP explains. That's what I do.

Yes, for users who weren’t scared of .ssh and even edited its contents from time to time. Because these were just their files, not “junk from apps and configs that hurt my sense of beauty”. Those who want to live in a clean folder - mkdir it.

Probably not in the roots home directory? ~/me/ feels good.

I’ve been a big fan of XDG for well over a decade. But I am yet stunned by how many big name projects still ignore it.

XDG is a great standard, but it seems it’s not enough. Probably time to abandoned home and put all personal files somewhere else — a subdirectory of home will do. And no standard name — call it what you want so no one can hijack it either.

Does anyone know of any small library in JavaScript/python/go/rust or any language that people use to write CLI apps that makes it trivially easy to implement this spec, even better if it also supports similar conventions for MacOS and Windows?

https://docs.rs/dirs/latest/dirs/index.html

The Windows equivalent madness is all the virtual folders for Documents, Downloads etc. that show in explorer without the full path (so no copy/pasting that path for you sir!), and may be mapped to some other place and suchlike. OneDrive takes over in a way that I always uninstall it and forgo the 1Tb of storage they give me for free with O365!

As well as some stuff dumped in the home directory too. The defense on both Windows and Linux is to just keep your own documents elsewhere, or at least in a subfolder of the home directory.

There are shell scripts like xdg-ninja that can help with this:

https://github.com/b3nj5m1n/xdg-ninja

> a Desktop directory. That last one has been created by Steam, which is quite unfortunate as I simply do not have a desktop or a desktop environment on my machine.

Chrome and a number of other software insists on creating some of these directories (e.g. Downloads) too.

However you can fix that by setting XDG_DESKTOP_DIR and XDG_DOWNLOAD_DIR in ${XDG_CONFIG_HOME}/user-dirs.dir to something else (e.g. your home directory).

I have tried everything under the sun, including user-dirs.dir:

    % cat ~/.config/user-dirs.dirs
    XDG_DESKTOP_DIR="$HOME"
    XDG_DOWNLOAD_DIR="$HOME"

Yet still:

    % ls -ld ~/Desktop
    drwxr-xr-x 2 mt mt 4096 Oct 22 18:04 /home/mt/Desktop/

Sometimes there's a ~/Downloads directory too. But not today.

I have no idea what keeps creating these. It's pretty annoying as I have a lot of intentionally short-named things in my ~ for easy shell usage, and it really sticks out.

inotify doesn't support reporting the PID of the process that triggered the event, but it seems fanotify does. Maybe I should set up a watcher for this.

try

    systemctl enable --now xdg-user-dirs-update.service

and just to be safe I have a snippet that parses that file and exports the variables in my login shell like .profile

(fish in this example but would be similar for other shells)

    sed -nE 's/^([^=#]+)=(.*)/set -gx \1 \2/gp' <"$XDG_CONFIG_HOME/user-dirs.dirs" | source

Try to set enabled=False in ~/.config/user-dirs.conf

https://www.unix.com/man-page/centos/5/user-dirs.conf/

Nix + home-manager. Boom.

Can you elaborate? How does that solve the problem?

nix home manager is very good.

The idea is that instead of creating a bunch of dotfiles and having to read a bunch of documentation, one declaratively states which features they want, say, for zsh or vim or git or xdg, etc; and then home-manager will create and manage them (typically they live in `$HOME/.config/whatever`).

https://github.com/nix-community/home-manager. Some examples on GitHub: https://github.com/search?q=filename%3Ahome.nix.

You are not gonna like the solution to that problem: Windows Registry like configuration file.

Centralization solve the problem in this case but have the downside that the central configuration file will become bloated and requires more memory in the long run.

Why do they have Steam installed if there is no desktop environment?

Many game servers can run natively on Linux and are still installed and launched via Steam.

Also there are desktop-less yet graphical ways to use Linux, such as tiling window managers

Looks like the python code has a bug, absolute paths are rejected.

``` if config_dir == '' or config_dir[0] == '/': ```

This problem can be solved by simply not storing anything in the home directory. Kepp your files in a separate directory like /data or /files and leave home directory for configs.

Storing your files in /home is also bad because usually /home uses Linux-specific filesystem and you won't be able to access your files from Windows. Storing files in NTFS partition solves this problem.

What if another user wants to keep their personal files on a computer too? Create /data2? This is what /home/$USER/ is for.

Also, /home may be mounted anywhere, including some NTFS partition.

Surely config goes in $HOME and thats that? I dont see how specifying another directory makes life any easier.

The article lists several advantages.

If you don't like having a separate directory for config files you can set your XDG variables appropriately.

While i agree dotfiles aren't the best, i'm concerned about creating too many alternatives. Then we risk ending up in this situation: https://xkcd.com/927/

As someone who uses linux to get other tasks done, rather than as a hobby in itself, I would rather have an aesthetically sub-optimal standard than have no idea what's going on.

what other "alternatives" are you speaking of on Linux?

This isn't some kind of new stuff. I'm not quite sure how old the XDG BDS spec is, but it's been at least 15 years; and the broad notion of using ~/.config rather than dumping stuff directly into ~ is older still.

Reply on news.ycombinator.com