Debian's /tmpest in a teapot
lwn.net
lwn.net
The XDG base directory spec (https://specifications.freedesktop.org/basedir-spec/basedir-...) was supposed to solve all of this...
In my personally opinion the reaping period is way too long, especially for /tmp/ as only few people really have an actual uptime that long but it does make somewhat sense for /var/tmp/ as it currently is "persistent".
Small you say? The block size in that device is the size of the memory page: 4K by default. That's 8 times the very typical 512b for persistent storage.
I've been bitten many times when trying to start in-memory only VMs with a disk image that measured smaller than available memory only to run out of memory on boot :)
Let's say it's not really for small files. It's for few files. ;)
I'm OK with a RAM-backed /tmp getting cleaned every boot simply because that's part of the technical tradeoff, however "on boot" is a bad trigger because its behavior is unreliable and can easily backfire and harm a user.
A classic example would be if the power is interrupted and the computer restarts. Unless the user is present and savvy enough to block it somehow, all of their short-term "throwaway" project/data just got deleted before they actually finished with it.
In other words, the difference to users between "I won't need this after tomorrow" versus "I won't need this by next boot" can be very important.
That's on them.
I use /tmp like this, because I'm lazy and don't want to have to mess with a directory I need to routinely clean up, but I know that if the computer crashes, it's gone. That's on me.
If you don't like this risk, you need to work in persistent storage.
/tmp is for programs' temporary files. Manually using it like this is abusing it. That's fine, it doesn't hurt anyone beside the user doing it, but they need to bear the consequences of doing it.
Which is literally what /var/tmp is and what the parent is advocating for.
However, clearing /tmp at boot is not the greatest technical choice. There's nothing really special about rebooting the system. The idea that files in /tmp are kept around for anywhere between minutes, hours, and days makes for bad policy which systemd-tmpfiles fixes quite elegantly. You shouldn't lose your temp file after 30 seconds because your machine lost power. You should be able to say "files created here are cleared after $short_time of inactivity."
IMO setting the expectation of 'wont survive a reboot' is more consistent. but even that fails with private tmp files that could be tied to ephemeral units
In Debian /tmp was backed by the disk. Trixie will come with RAM based /tmp. Update is pushed.
That being said, all my systems have /var/tmp as just a symlink to /tmp, regardless of what the FHS may have to say about it -- and it's cleaned weekly, not just on reboot.
[0] https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.htm... is the current home, apparently
Or it's useful system behaviour.
To-MAY-to, to-MAH-to.
> […] should exist under a proper /var/{cache,lib} directory.
Are these directories available to regular users like the tmp dirs are?
Yes, system users tend to not have a home directory. But you are fully able to create one for a new system user if this is what you want to do.
You are also able to create all kinds of temporary directories and read-only filesystems for a process (e.g. using systemd) if this is what you want do to.
Yes there is a lot of fragmentation and no real standard, which is a little disappointing, but you sure have options to work with. And these options do not have to involve any legacy-ish filesystem components like /var/tmp that nobody knows the purpose of.
The purpose of /var/tmp is temporary files that should survive a reboot. A potential example where that would make since is a partial download file. It is only needed temporarily but if a reboot happens while the download is in progress, you probably don't want to have to start over.
Also if /tmp is or might be a tmpfs, it is probably desirable to use /var/tmp for larger temporary files that don't fit in RAM.
/var was originally for network booting machines. / is read-only fileshare and /var is for user storage. /home -> /var/home too. It's campus wide workstations installation type of thing.
So, /var/tmp is kind of wrong, and so is /var/log/messages on a laptop, and besides the entire /var is useless on most systems. But it's not a huge waste of resources either, so it's left as it is along the rest of directories under / for years.
/var wasn't originally intended to be a system directory, it's just a variables directory. Almost by definition it's writable.
I have run (Sun) workstations off of NFS, including completely diskless ones. /var was often done on a per-workstation basis.
(I'm actually planning out a diskless HPC cluster that will be deployed in the coming months.)
Honestly, we should probably abandon the hope that we will ever get full universal adoption. It stinks even worse with the half-adoption state we are in now, with some apps writing to those directories and other apps not. If it wasn't for arch Linux folks trying to drive apps to adopt it more, I don't think it would be. Hell, default Debian/Ubuntu still doesn't set XDG_* directory ENVs by default (some flavors do), and some apps ignore the prescribed "if not set, use this default" nonsense in the spec and do what they have always done because it doesn't break compatibility for users.
Part of the spec stinks, too, like the spit between cache/config/data config where the spec has no written rules specified on when anything is expected to be cleaned up and when or what can or should be backed up by the user.
Let's move to containerizing most apps, putting everything in little jails so we don't have to deal with where they want to write to. They can have their own consistent and expected behaviors on their own island. Snapcraft, flatpack, or any of a bunch of other solutions are already available to solve this problem. Don't have to worry about what some app left behind or wrote to your system when it's all contained.
You are supposed to leave them unset if they match the defaults.
Programs ignoring the spec is another matter and should be fixed in those programs not by needlessly bloating the environment of every process with meaningless data.
Things like .git and .ssh not are not meaningless data even though they are not in ~/.config/ssh/ or ~/.local/shared/git.
It stinks about half as much as before.
> Part of the spec stinks, too, like the spit between cache/config/data config where the spec has no written rules specified on when anything is expected to be cleaned up and when or what can or should be backed up by the user.
Caches don't have to be backed up and can be deleted if you need space. Most people will want to back up configs. What other data should be backed up is a question different people will answer differently. And containers don't solve this.
Otherwise, you're directly heading for a nasty surprise at the worst moment.
n = 1, but, uh, this is one major reason I don't use Linux. It really bugs me.
Actually AppData can get a bit polluted too.
Typical of our society, we want to make everything someone else's problem.
I guess we deserve some of the SaaS enshittification and it's spending obesity ;-)
Yes, this.
> Typical of our society, we want to make everything someone else's problem.
I don't think that's a fair analogy.
It's more like moving junk from the living room to the attic. I'd prefer to have a clean living room and a messy attic.
> Yes, this.
Like "My Documents" (or "Documents" nowadays)? Now that's some dumping ground.
~/Library also isn't something that is used to be; it was good idea, but mostly ignored. Basically similar idea as XDG_(CONFIG|CACHE|DATA)_DIR.
What's worst, is that many projects ignore any effort to move them to XDG schema. They take the approach, that they always did it this way, it works, so why changing it.
Dot files in my homedir on Sonoma machine:
.CFUserTextEncoding
.DS_Store
.Trash/
.Xauthority
.amt
.android/
.anyconnect
.bash_history
.cache/
.cargo/
.config/
.cups/
.degit/
.deno/
.docker/
.eclipse/
.emacs.d/
.gitconfig
.gitignore_global
.grass7/
.hex/
.homeassistant/
.ipython/
.java.login.config
.jupyter/
.kube/
.lemminx/
.lesshst
.local/
.matplotlib/
.mix/
.mume/
.node/
.notable.json
.npm/
.nvm/
.oh-my-zsh/
.oracle_jre_usage/
.pip/
.profile
.psql_history
.python_history
.rest-client/
.rustup/
.sqitch/
.ssh/
.subversion/
.swiftpm/
.thumbnails/
.viminfo
.vscode/
.wget-hsts
.wine/
.yarn/
.yarnrc
.zcompdump
.zsh_history
.zsh_sessions/
.zshenv
.zshrcCleaning up /var/tmp on a timer is relevant to this academic environment (desktop-based research computing):
1. Each Debian machine is used by only one graduate student, but students do not have root access.
2. Today, /var/tmp is the only persistent local directory where the student has write access ($HOME is on a network filesystem backed up by the university).
3. Within the student population, there is strong institutional memory that /var/tmp isn't backed up by the university and isn't extremely robust (e.g., RAID), but also that nothing there is automatically deleted.
4. Students use /var/tmp for hundreds of Gb of data from simulations that take days or weeks. $HOME is too small and too slow for this.
5. In practice, less than 1% of students lose data through disk failure, accidents, etc.
6. A much larger fraction of students will lose data when sysadmins, who didn't get the memo about the /var/tmp change and thus haven't addressed the ingrained institutional memory, deploy new Debian machines.
7. Some of the students who lose data won't graduate on time.
How is that not a site specific problem they introduced? There's no standard pathname because there are valid reasons why you want to have nothing on the local system, such as when it doesn't have a local disk.
If you (read as "the admins") have configured the system where there's very few acceptable locations to store local data, and that's a specific need, then it's up to the admins to provide a solution to the problem, not the Debian distro, which is flexible enough to handle this fine.
It's not that hard to create your own location for data if you need to, and symlink to it from other locations if required. Need a /tmp/$USER to be a local disk location and /tmp is a ramdisk? Create a script that sets up the correct local location and created a symlink to it in /tmp and make sure it's run on boot and maybe once daily. Both cron and systemd solutions could work for this. Worried that user tmp files will be cleaned out when they shouldn't be? Put in the correct exceptions for tmpwatch or whatever debian uses, or disable that service.
Not only is this not rocket science, it's literally the job of the admins that design the OS deployment so that it works for the intended use cases.
Wouldn't that be better than trying to get an upstream project to preserve an informal social contract that students inadvertently relied on?
They could try to, sure, but normally local admins in the unis don't cave in to any demands from the students. Sometimes they don't agree even to the demands from the faculty!
Yep can confirm; My uni's IT department recently began blocking all inbound ssh traffic for the entire university network (including the data center ranges), and shot down any requests from students, faculty, clubs, and enterprises that asked to have an IP or two whitelisted so they could access their infrastructure from off-campus (except ITs own services were whitelisted; can't be inconveniencing them now)
A subset of us that got refused formed a mini anti-IT 'cabal' of sorts, eventually found an oversight in how they implemented the block (it just pattern-matched the initial ssh handshake version string; you can change it by compiling openssh from source), and have since been on our merry way, with IT none the wiser.
But hey, at least the security guys can sleep soundly at night thinking we're still being inconvenienced by their arbitrary decision. Clearly they must think everyone is as incompetent at locking down a network's security as they are.
Teach students about backups and make network storage available for the code and recoverable checkpoints to restart failed simulations/computations.
If reliable, available storage isn't available for all students to graduate then allocate more money to technology resources. Speed is much less of an issue if it's only used for periodic checkpoints.
If you create an anonymous temporary file, then it will be deleted when the last file descriptor to it is closed-but anonymous temporary files are difficult to use (even given access via /dev/fd aka /proc/self/fd). You can delete it using atexit() but that doesn’t get run if the process terminates abnormally (core dump, kernel panic, kill -9, power outage, etc)
Maybe something like a pidtmpfs where the top-level directories are pid numbers, you can create any files or directories you want under them, but it all disappears as soon as the process with that pid exits. Actually, pids are a bad idea due to pid reuse, you want something like a process serial number or a process UUID which doesn’t get reused.
Another somewhat more flexible idea would be that you can create arbitrary top-level directories, but they - and all their content - are automatically deleted as soon as the last open fd to them is closed. That way you could have multiple processes sharing a temporary workspace, and the workspace could survive the failure of any one of them, but if they all fail it gets deleted
Too bad that unprivileged namespaces have been buried by the big-name distros, though.
It looks like Debian used to have them disabled, but then re-enabled them for the sake of web browsers [1]; I'm sure they'd re-disable them if they found some solution similar to Ubuntu's AppArmor one. For other distros, it's difficult to find up-to-date information on whether they enable or disable unprivileged user namespaces, since many have flipped back and forth over the last decade.
All that is to say that unless your program is given privileges itself (e.g., Docker), or can wheedle user-namespace permissions out of the packagers, there's no chance you'll be able to distribute namespace-using code and have it work consistently for most users.
[0] https://discourse.ubuntu.com/t/ubuntu-24-04-lts-noble-numbat...
[1] https://salsa.debian.org/kernel-team/linux/-/commit/a3819178...
https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged...
[0] https://salsa.debian.org/kernel-team/linux/-/commit/a3819178...
Consider this use case: I have a test program which executes some unit tests, it creates some temporary files. I want the temporary files it creates to all be removed when it exits, even if it core dumps. But while it runs, I want those files to easily be accessible to subprocesses it spawns. I don't think systemd addresses this, because my unit test runner isn't going to be a systemd unit.
So really I was talking about a generic facility an arbitrary program could use, not just something limited to systemd units only.
For this precise use case, what if you deleted the last run's temporary files at the start of the test program?
It even works when you write a temp script and execute it in /dev/fd/...
However since you can't control the file name in that case you cannot control argv[0] for such temp executables which may be a problem for BusyBox-style tools.
Have you encountered other issues with using that approach for named but auto-cleaned files?
Related example: testing a program which expects a file name to match a certain pattern; can’t do that because /dev/fd file names are just numbers
One case where I need an executable named temp file is when I need to pass that named file to another process which in turn will call that file (so I don't have control on how that other process will call exec)
Sadly you can’t exec an fd on macOS.
I don’t think Apple is going to change that, because I think it causes difficulties for their security/codesigning/sandboxing infrastructure, and I think they don’t see the difficulty and risk of making it work with that as being worth the rather limited benefit
It is used by all the application software for "fire & forget", so the app code can just generate the file and not have to hang around (and also, hang) while waiting to delete the file some unpredicatble time later. The file may or may not ever be used in some cases, and if used, it's used in some other process that the writing process doesn't know how long it will take unless you rig up some way to signal.
Every user of the special directory knows this special property of it, as this is it's entire purpose in the first place. It's not /tmp so no surprises. It eventually got used for all kinds of other things too because it's just so handy.
It works best with atime, which means mounting the filesystem with atime enabled, which I don't like to do in most cases, so ideally you make this it's own tmpfs in ram, and then the atime doesn't hurt much.
But it also works well enough with just ctime & mtime and a slightly longer TTL.
Even if a file is large and the user is on a slow link and the file is still downloading when the TTL expires, and the filesystem does not have atime, and so the reaper does actually delete the file while still in use, it's still ok most of the time even then, because the actual process that has the open handle still has it and can continue downloading the file until they close it, even while it has disappeared from view for the rest of the system.
But with atime, it's like magic, the file just naturally lives exactly as long as anyone is interested in it, and gets reaped 2 minutes after that, with no application process needing to keep track of it. The reaping happens regardless of crashed processes or reboots, graceful or ungraceful, etc.
It's something I did almost the first week over 20 years ago at a company I worked at from '99 to about '22, and got used for practically all temp files, and the normal tmp became the special case you only used in special cases. (actually the application software really never used the global /tmp, there were various other customer-specific and app-specific dirs)
Basically it was like an OS feature (for us) in that every system (of ours) always had this magic tmp dir.
This worked fine on old sco systems without even gnu find with it's handy -delete, let alone cgroups or even a ram fs. In fact it started there.
/var/tmp being cleaned on boot is probably fine despite contrary standards in the past.
But automatically deleting files - in either of those - while the system is running is going to break all sorts of things.
As a random example, `chromium --temp-profile` will have half the profile deleted (since some particular files aren't accessed frequently) while the browser is running. This looks designed to cause system-level UB.
Note that locks on directories are pretty rare; locks on a file within the directory are the de facto standard, on the assumption that only cooperating programs will access them.
Hanging my comment here. Sorry to slap it into your universe.
My knowledge says... is it fine?
My RAM is for applications, and I buy RAM accordingly.
Most in this discussion seem to miss something quite vital and important. Linux is excellent at caching most-needed, most-used data in RAM, eg buff/cache in 'top', and that should be given higher priority than some temporary file a user might slap in /tmp/.
50% of my RAM, randomly flushing out buffers and cache! Buffers and cache, which stores commands I use frequently, libraries often used by applications, and so on! Any server that is reasonably loaded, will show RAM fully used by buffers/cache, excluding active RAM requirements. Now I'm to give 50% of this up, having those buffers/cache flushed for... someone unarchiving a tarball?!
And my swap file, something for emergency application RAM usage, now being stolen by /tmp. We're basically taking the most expensive storage space on a system, RAM, and relegating it to... a few log files, someone untarring software, and also...
Hard limiting its size to RAM requirements?!
This is sheer madness. It smacks of people with desktops, with enormous amounts of free RAM, making decisions for servers.
This change is bad.
Thoughts:
* It is not relevant what systemd does, endless things in Debian and other distros diverge from what systemd does
* Debian isn't aligning with 'other distros' when doing this, as there isn't a consensus here. Debian isn't some hold out, in fact there are far, far fewer installs elsewhere doing this.
* Stealing my RAM for transient files is dumb
Nope, unarchiving tarballs doesn't write the tarball content to /tmp.
> It smacks of people with desktops, with enormous amounts of free RAM, making decisions for servers.
* Servers usually have way more RAM than desktop devices? Unless you mean VMs, in which case the VM can't decide what device /tmp lies in anyway.
* in any case it will be trivial to just not use it for your servers, if you prefer /tmp to be on disk. Just like I already used tmpfs for 2 decades in Debian, it is a single line in /etc/fstab either way.
My whole post talks about putting files in tmp!
And this change is a performance killer for that reason. Taking all that juicy, super fast RAM away from system caches and buffers, which yes is a performance killer.
You want sensible defaults. Telling people they can change a dumb default to something smart, isn't a viable response.
I mean my god, you say you've been using linux for 20+ years, so you surely know, doing this for a performance gain is absurd! We have mega fast SSDs, and an elegant filesystem layer, writing to disk is immensely fast compared to spinning disk days.
We gain almost nothing here, and lose RAM caching of libraries, data, buffers, which the kernel is very, very good at these days too.
It's a dumb default. Immensely so! And statement like "you can just" aren't the point.
This slows almost every single system down.
You know, this could be done right. Instead of stomping on everyone, just create a new tmpfs for just this purpose. Let people use it if they want.
And why on earth suddenly start forcing file cleaning. Debian has not had that default in decades, changing it now because someone else called redhat does it, isn't a valid excuse.
Debian doesn't need to be more like redhat.
Detect the process that created a file and delete it 24 hours after the process ends.
Systemd now essentially control Debian both the OS itself, and though Boccassi, the development.
This can pose problems. Boccasi steamrollered, but relatively politely.
People were more interested in bitching about edge cases where maintainers / projects / authors were doing things they should not have been doing in the first place than they were about addressing those issues, so he addressed them himself.
The man literally put up when others would not shut up.
But perhaps 30% could be in what you call "edge case", since your statistics is entirely based on yourself.
Good work pushing through it Boccassi!
Why not just have the installer _ask_ me what configuration I prefer? Is there some reason you have to force the change, announce a victory, then make me to go mucking with "defaults" after the fact?
You will have to disable it separately. From the article:
“To stop periodic cleanups of /tmp and /var/tmp users can run touch /etc/tmpfiles.d/tmp.conf.”
Because nobody bothered to actually put in some work to implement that. As I've said on the ML, if somebody does the work, I'll review it. But it's one of hundreds of different settings, and it's obviously not worth anybody's time to do this work, as it's largely inconsequential and trivial to configure via the supported config files. Complaining on social media is of course cheap enough that we get plenty of it.
Likewise if you make /akirastmp, the OS will leave it alone, but using the directories the OS clearly makes temporary for things that aren't should be the configuration to require manual setup since it's the unusual one.
(* Council's licensed private operators)
It's the Poettering way. And GNOME, for some reason.
I put it down to the level of monopolization of industry and to a certain extent our culture. These people mean well and they're mostly just reacting to a really baffling labor market in the face of a failed and entirely captured internet revolution.
With the right pair of eyes you can look across the dunes of github into the early 2000s and see where open source culture peaked, crashed, and rolled back.
Some people will see their stuff vanish from /var/tmp and that will be their day of learning. I guess they won't have a backup of files in there either. They were one misfire away from loss anyway.
Sure it feels different when the distro just ran over your misconceptions. On the other hand, when I encounter a stray tmp dir, I assume everything in it can be deleted with negligible consequences. So I tend to rm them without even a glance inside.
If half a dozen people in your town of 10,000 use bins for storing soup, but one year the council procures bins with holes in the bottom to keep rain from collecting in them (making them hard to move, and also providing stagnant water for mosquitoes, ie something that benefits nearly everyone)...
...what would be your opinion of a blitzkrieg of online and media criticism of the council for not considering the needs of soup binners...by people who do not store soup in their bins but are screeching about the needs of the people who do and they say will be affected by the change?
And then someone on the council has said "alright, fine, look...I doubt there's that many of these people. I'll go talk to them, who are they?" and the screechers admit they don't actually know any soup binners, so the councilmember looks them up and goes to the house of every soup binner and either plug the holes in their bin for them, gives them proper soup pots, or prints out a list of soup pots they can find on amazon...or finds out that they say "well gosh I had a soup pot in the attic but I never got around to using it, I'll just switch. Thank you for the heads up that the new bins will have holes in the bottom."
Now, auto-cleaning /var/tmp... that's going to cause SO MUCH data loss across people I know. I see a parallel with keeping files in your Desktop: yes, that's not what it's there for and ideally you'd store things properly, but you don't go around deleting people's documents just because you don't like it!
Time to start sending warning E-Mails, I guess.
I symlink some of the desktop cache files to /var/tmp, but since that never got deleted for me (on Debian and Manjaro) I also started linking things like my browser profile there. I kind of knew that /var/tmp may not be the correct folder for it, but if it works, it works you know.
A bit of background on my setup, for why I even want to create such symlinks. My home folder is on an automatically mounted USB stick that I transfer between systems (laptop and desktop). On shutdown, I rsync the home folder to a local backup folder, which gives me a small distributed backup system. Using the same home directory between systems with different distributions and CPU architectures (I'm using the pinebook pro arm laptop) works surprisingly well. But I'd rather not stress the USB stick with a bunch of frequently accessed cache files, hence the symlink. I end up also symlinking the browser profile, because that caused version incompatibilities between systems, and I'm ok with having local browser settings/history on each device.
As an antidote I had a hard time for a few years after I stopped using irix, I wanted(or at least my fingers wanted) the home directories to be under /usr/people
Wait, what is Desktop for, then? Application shortcuts? We already have "pin to taskbar" for that.
The original Mac Finder even had a "Put Away" command to support this workflow.
I believe this idea partially arose by analogy with physical workflows, and partially as a way to more easily work with related files spread across multiple floppy disks: desktop items remained visible when a floppy was ejected, and the system would prompt for the floppy by name if you attempted to access a file on an ejected floppy (ejected floppies themselves remained visible on the desktop until you dragged their icons to the trash).
[Pedantically, the original (System 1.0) Mac Finder had a more general, slightly buggy "Put Back" command, removed in System 2.0, and finally replaced with the "Put Away" command in System 2.1. Source: past experience verified through testing at [1]]
> The /var/tmp directory is made available for programs that require temporary files or directories that are preserved between system reboots. Therefore, data stored in /var/tmp is more persistent than data in /tmp.
> Files and directories located in /var/tmp must not be deleted when the system is booted. Although data stored in /var/tmp is typically deleted in a site-specific manner, it is recommended that deletions occur at a less frequent interval than /tmp.
* https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.htm...
IRIX docs:
> The directories /usr/tmp.O, /var/tmp, and /var/spool/uucppublic are public directories; people often use them to store temporary copies of files they are transferring to and from other systems and sites. Unlike /tmp, they are not cleaned out when the system is rebooted. The site administrator should be even more conscientious about monitoring disk use in these directories.
* http://rsusu1.rnd.runnet.ru/sgi/advanced/ch8.html
From FreeBSD 1.0 (July 1991):
/var
[…]
tmp/ temporary files not removed between system reboots
vi.recover/ recovery files for the vi(1) editor
This is also true for Solaris, which I used to admin many moons ago. So I'm not sure where the idea that /var/tmp gets cleaned on reboot came from since I have always understood it to be fairly static.
In OpenBSD, /var/tmp -> ../tmp and /tmp is cleaned periodically and not preserved across reboots, but there are some specific exceptions and you have to dig into /etc/daily and /etc/rc scripts to know what they are.
Now they will either fill up my RAM or require a large swap partition (which I usually don’t have as it’s otherwise wasted space).
I really like making /tmp the default destination for downloads, so either I need the file and move it elsewhere, but usually it’s an archive and I just want a file from it. Saves me from an ever growing Download folder.
Those files are most dispensable and should not consume precious RAM.
Some file systems are even adding specific tmpdir support where fsync() is turned into a noop. So they have all the advantages of tmpfs without eating into precious RAM/Swap.
I don’t care about those files once I’m done with them - so far they would only be gone with a reboot, which aligns well with my definition of done.
Sure, /tmp is cleaned on boot. But if the host is rarely rebooted it’s pretty inconsiderate to just leave enormous quantities of unneeded crap lying around in shared directories.
/tmp isn’t supposed to be a garbage dump. It’s supposed to be for data that’s actually in use but transient or relevant to running processes. You’re still expected to be a good citizen on shared hosts and clean up after yourself when you’re done.
The reason I’m doing this in /tmp is that I can just shut down the computer and have all those staging files gone.
(Same reason I shut the computer down at the end of the day instead of suspend - I want to start with a clean slate the next day)
But it shouldn't be too hard to write a relativly simple systemd.unit file that does that at boot. After all the main part would be `Requires/After=local-fs.target` and something like `ExecStart=bash -c 'rm -rf /var/tmp/*'` I think (you'd need to double check what exactly to do if you want to do this).
Defaulting to /tmp in tmpfs means I can remove a line from my post install setup script. That doesn't matter much. But it also means less divergence between my system and the default, so there should be a reliability improvement from other people stumbling over problems with the ramdisk before me. So a win all around, happy to see it.
I've never thought about mounting a tmpfs folder for compiling stuff, but that's actually a pretty good idea.
Using tmpfs (or RAMDISK) for compiling and discovering that it’s not actually faster is a virtually rite of passage for every developer. :)
If you do try it, at least benchmark it. You’ll probably discover that you spent more time setting it up and copying files around to volatile storage than you’d ever gain back via minuscule compile time speedups. Compiling is not IO bound and your files are already being cached in RAM by the OS after the first read.
You shouldn't need to copy files to volatile storage, just storing the outputs there. Maybe only storing the intermediate outputs there. Some build processes just dump intermediate outputs all over the place, but a lot of them have a directory for those, and it shouldn't take much to make that directory be on tmpfs or similar.
I have a tmpfs folder mounted for my downloads folder for my browser for similar reasons; not for speed but to reduce drive wear and automatic cleanup so that it doesn’t pile up for forever.
What are you compiling that’s actually faster in tmpfs?
A decade ago, people were surprised to discover that going from HDDs to SSDs didn’t impact compile times very much. The files are quickly cached in RAM anyway and the bulk of build time is CPU, not I/O.
Going from NVMe to tmpfs feels like an even smaller difference.
This is one reason I like Arch. It is almost as vanilla as can be (some exceptions may apply) while still being a "build it yourself Lego set" (build not in the compile sense).
Arch, Gentoo, and NixOS Minimal are all pretty wonderful in that they give you the tools to build whatever system you want, without a bunch of extra crap that you'll never use.
Also the tone of some of the included quotes is so offputting. Their statements and arguments for this change don’t even acknowledge the fact that there are different views on this. This change comes across as progress for progress’ sake. Makes me want to make a 500TB /var/tmp and never delete anything.
I swear I remember back when I first switched to linux (Ubuntu) back in the late 2000s, this was how it worked: Instead of a tmpfs or automated wiping on boot, a system crontab entry that would use "find" to delete files last accessed more than some time ago (a day? a week?).
Except for the risk of filling the disk if files are created too quickly, this would seem to be the best of all worlds to me - stuff can't stick around forever unless it's constantly being used. A download you access once would stick around for a day, while a heavy project that's using /tmp/ would stick around for the week or month you're using it, even across reboots, then get deleted later.
Care to share? Looking at https://systemd.io/TEMPORARY_DIRECTORIES/ and I don't see any description of how to change the default behaviour dictated by systemd.
How do I disable the automatic file reaping? How do I configure the maximum age of temp files? Neither are mentioned. The bulk of the page is just a long list of passive-aggressive suggestions about how users are doing things wrong.
> Users who want /tmp to remain on disk can override the upstream default with `systemctl mask tmp.mount`. To stop periodic cleanups of /tmp and /var/tmp users can run `touch /etc/tmpfiles.d/tmp.conf`.
I have some systems (usually VMs) where RAM is much bigger than the root disk. For example, 500+ GiB of RAM and only 100 GiB of disk. I have other VMs where the disk is orders of magnitude slower than the RAM (e.g. EBS on AWS EC2).
I wonder what I should be doing for swap and vm.swappiness. Usually I just run with either no swap or a tiny 4 GiB swap file on /.
On small memory systems, it really is useful to have 2x ram, because it can be pretty useful sometimes. On large memory systems, it's highly likely that your system is too far gone once it's used a significant amount of swap, so 512 MB seems like as good a place as any to stop. Swap used %, and swap in / out rates are a very good measure of system health, and if you have no swap, you lose out on those metrics. Ideally, for a small leak, you'll have enough time to get an alert about high swap use, and come in and inspect the situation before you start getting OOMs, but for a large leak, you want the system to fault quickly --- it doesn't do anybody any good to be up but deathly slow due to thrashing for a long time.
These days I set up servers without swap, and I have not run into any issues doing that.
I always heard that the "2x RAM" thing came from some primordial BSD release.
I have literally never wanted this and so I'm curious why you do. This is why earlyoom is a thing precisely because grinding your system to a halt is often worse than killing and restarting some processes.
High swappiness so you run slower but never lock up or low swappiness with an aggressive oom killer so you don't lock up are to me the only sane options. Why do you want your servers to be up but thrashing?
Also make -j is prone to run out of memory if you have many cores, I’d rather adjust the number of threads than have my system grind to a halt.
I wish it worked like Windows - I've never had to reboot due to lack of RAM there, and even when the system is unresponsive ctrl-alt-del is very reliable.
It happens, albeit quite rarely.
And anyway, I don't recall having to tune Windows to work properly.
On an unrelated note: if you're the DrMcCoy from Twenty Sided and GOL it's nice to see you.
> Red Hat Enterprise Linux (RHEL) and its clones, as well as SUSE Linux Enterprise Server (SLES) and openSUSE Leap, still default to /tmp on disk.
So at least some commercial OSes, if you will agree that we could refer to the first two of these distros as such, are also staying with old decisions for a long time.
And even though RHEL in particular is know for its insane dedication to keeping API compatibility with backports of software, they still are in a position to change things around between major versions. And I believe they do do that quite a bit, which is why they also give so long time before EOLing old versions so that enterprise has plenty of time to adapt to the changes with major version upgrades.
(And the average developer has enough savvy to change settings).
... also, most of the compilation tools I use end up dropping their intermediary files in a peer directory to the source tree or a build directory at the root of the project, not in /tmp. I'm not sure what tools people are using that are leaving compilation intermediates in /tmp.
Unfortunately tmpfs is not so good when the data does not fit into RAM. The first thing that happens is it writes the overflow to swap, which is much slower than writing to disk in the normal fashion so you lose the speed advantage. If you store so much you run out of swap the second thing that happens is your system dies.
This proposal comes with a bigger hairs. The issue is you can't trust all apps with clean up after themselves, meaning they will exit leaving crud in /tmp. That means if the system isn't rebooted /tmp tends grow slowly as this crud accumulates, which eventually means even in the best case the system will become unstable after enough time. Their solution to that is to just delete old files on the assumption they are crud. But that's a kludge as there is no way to be certain a file is crud of not, and deleting files that will be re-used makes the system unstable.
Which in the end means it depends on use cases. A typical desktop user with lots of RAM (at least 8GB, preferably 16GB) will probably see a benefit as their files in /tmp will fit into RAM. Typical means they don't do something that eats RAM and disk like editing video files, and they reboot their system occasionally. Other users will lose because tmpfs overflow to disk so their system will at best be slower, possibly unstable if they didn't know to allocate lots of swap when they installed the system.
Notice servers don't neatly fit the typical use category, and also notice variants of OS's that target servers don't use tmpfs for /tmp. Debian doesn't have a variant that targets servers, but whether /tmp uses tmpfs is easily configurable for sysadmins. Editing text files is their day job after all. In fact that ease of changing the default was the main argument for the change on the Debian lists. The end result is the change won't overly effect sysadmins of Debian servers either - it's just one more thing they have to change in what is already a long list.
TL;DR: server admins won't care about the change and typical users with big laptops doing normal stuff will be happy, but normal with cheap laptops or are doing unusual things will have their world turn to shit if they aren't comfortable with the command line.
That keeps the file in the page cache until it’s evicted, but instead of being written to swap it’s written to the fs. With the swap partition usually being quite small and the fs today being all the storage there is on a personal machine.
The upstream defaults for systemd are to mount /tmp as a tmpfs and delete files that have not been read or changed after ten days in /tmp and 30 days for those stored in /var/tmp.No I don't know why either, while not the stupidest thing I have seen, it comes close.
If they want to do this they need to modify tmpfs to do a better job of shifting data to disk when full, instead of letting swap handle it.
But more importantly, don't write large files to /tmp/. Back in the day we gave /tmp/ its own partition just so those files didn't fill up the root mount. Write big files there and it would fill up. Plus, big files benefit more from a faster drive, and back in the day we used to put the root mount on a slow disk, and have a fast big data disk which is better for large files.
I think an option to prevent tmpfs from getting shifted to swap (or disk) at all would be great.
I did this on Fedora, and it left a bad taste in my mouth.
Ah, yet another group of Linux users looking to get ssh to adopt the XDG specification. Moving ~/.ssh to ~/.config/ssh has been repeatedly requeseted. XDG_RUNTIME_DIR is going to be even harder because it can be nonexistent on BSDs.
I mean I could start using /var/tmp or make a directory in $HOME and clean that up occasionally, but that would mean I have to change old habits. ;) so I guess masking tmp.mount it is.
If we're doing because the systemd maintainers say "we have to do it because systemd", though, well...
How long till the systemd guys make package dependencies and installation as a service? You know, sponsored by the IBM/Redhat team, and signed by Microsoft?
signed,
a Devuan devotee
Hopefully the new Debian version of this script is a bit smarter!
On the other hand, a sysadmin strives to be lazy, not by postponing problems, but solving them the appropriate way, and preventing it from happening again.
No! That's what quotas are for. Everyone should keep track of it's own garbage.
My goto commands "df -h" and "mount" are cluttered with all kinds of ephemeral filesystems, with no easy way to filter (say, a one-character flag). lsblk is close but .. lvm and snaps.
The output of df -h looks pretty sane and understandable.
I see some entries for /dev and /run and /, some stuff for EFI, a bunch of mounted ZFS datasets, and it's all very readable with no line noise in any column.
---
Now, that said: I have some docker containers. My own previous goto command of mount reports a complete clusterfuck, and all of the clusterfucked lines are due to docker doing docker stuff.
I don't know how things like 4UYSHHPFBZQNFZ9Z698JDLUZWJ, LK33RZTN5ZNFGPJNDUHM0HTSD1, 2G0A4YYUVCMP9HZSM5UXOJW0BX, ZH29C5YNZHJ22QMY63EWV5RF3P, and 32XOCXBUY41QQJTPPA8HE6EVZ6 add any coherency or clarity[1], but I assume that since the computer quite easily knows what this shit means then it needn't be bothered with dumbing it down so that a human can parse it in any sort of memorable or repeatable or communicable fashion.
(Which is, of course, bad behavior on the part of the computer. A UUID is also awful, but at least it has some punctuation so that it can have some cadence if it must be dealt with as-is.
It's like we learned nothing from DNS[2], or as if translation layers can't exist even within a filesystem[3].)
1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2016788/
Huh? Someone explain this to me, wasn't fstab the mechanism to define things like this? Why does my init/service manager need to mess with it???
What’s more, on systemd systems, fstab is dynamically converted into native systemd mount units at boot, so in many cases fstab is purely a facade/compatibility shim over systemd doing all the mounting anyway.
While not terribly prescriptive, “man systemd.mount” recommends /etc/fstab be used to manage “mounts for humans” due to its simplicity and accessibility. So it doesn’t seem like fstab is considered legacy or deprecated by systemd; rather, this seems like more of a porcelain/plumbing distinction.
Naturally this leads to more difficulty in testing and maintenance over time. But that extra work pays for the benefit of having a very long lived product and compatibility with past integrations.
At the end of the day you have to decide if you're building a product to be easier for the users, or the maintainers. Personally I'm on the side of the users.
Laptops have ridiculously large amounts of RAM nowadays. My Linux laptop setup barely makes a dent in the RAM, even when running 3 different Web browsers and other gluttonous desktop programs. `tmpfs` is a great use for excess RAM, reducing wear on SSD. (I also disable swap.)
I've also done things like build an entire large ecosystem of packages in `tmpfs`, when the build server happened to have mirrored 10krpms drives, and I didn't want the tons of intermediate files to eventually be synced to disk. (Even though, with disk, they would also probably hang around in Linux filesystem buffers, not reads hitting disk each time.)
Edit: du -hc ~/.cache says it's 25G.
I didn't think of the massive ML models, because I store those in the home directory, where I can see them.
(Incidentally, since you mention Stable Diffusion: for an earlier version of SD, I went to some care to make sure I didn't accidentally lose the checkpoint file, because I didn't know whether it would be pulled from distribution. Then there were regressions in the SD training data.)
That works for most things, and that's how it should be. But sometimes the dev just doesn't provide links and you either have to read their code to figure out the model URLs, or simply allow the automatic download that ends up in .cache.
Your average person isn’t running Linux or compiling things. The comment was in the context of developer laptops.
(Well and Yocto builds, but that's not your average 'compiling things'.)
If something is supposed to persist past reboot, I wouldn't put it in `.cache`. Though I don't know offhand what the official documented behavior of `.cache` is, and I can't immediately find that documentation (maybe some open desktop cabal thing?).
> $XDG_CACHE_HOME defines the base directory relative to which user-specific non-essential data files should be stored. If $XDG_CACHE_HOME is either not set or empty, a default equal to $HOME/.cache should be used.
My ~/.cache could be rm -rf'd without too much worry right now, but that doesn't mean that persisting it isn't useful. For example on my system right now:
- Browser cache is useful to persist, especially with some larger sites.
- I put my Go module and build cache in ~/.cache, and while that can be deleted it's useful to persist because it make builds shorter, and avoids having to (re)-download the same modules over and over again. Note that at the moment my internet is kind of crappy so this can take quite a while.
- Some other download cache things in there, from luarocks, xlocate, few other things.
- I store psql history per-database in ~/.cache/psql-dbname. It's useful to keep this around.
- Vim backup files, persistent undo files, and swap files are stored in ~/.vim. The swap files especially are important because I want to keep them after an unexpected system crash.
Some of this is solvable by moving stuff to other directories. Others are inherently unsolvable.
I also have just 8G of RAM, which is fine but not fine for storing ~/.cache in RAM.
It's not going to cause "problems". It's just going to massively slow down many use cases that rely on downloads.
I used i3 and I absolutely love the way it feels. However I would like to configure it in a way that I can press a button and every app turns into dark mode. My main apps (Firefox, vscode) have an option to say "use system theme" i.e. if the os is dark, theme this app dark. Great! The problem is I don't know how to set the system theme. And every time I've asked around, I see people respond with GTK, QT and what not. I don't have those, I have i3.
Help?
gtk: hell if I know, the web sez there are css files, and there are probably some gnome tools to set the theme.
qt: no clue. and the web was unhelpful
native X11: now we are talking I know this one, X11 provides a database that applications can use for configuration and design. The main interface is the xrdb command. and all good application should include the pertinent points of their call tree to get you started in theming them. http://man.openbsd.org/xrdb
GTK and QT are UI toolkits and do control how apps render UI. Your apps will use GTK and QT even if you use i3 to manage the windows.
You might be confusing GTK with Gnome and QT with KDE, each of which has its own window manager.
> You might be confusing GTK with Gnome and QT with KDE, each of which has its own window manager.
Just cleared up a lot of confusing in my mind. Thank you.
https://askubuntu.com/questions/769417/how-to-change-global-...
I ran gsettings set org.gnome.desktop.interface gtk-theme 'Adwaita-dark' and firefox isn't in dark mode, despite the fact that i have firefox configured to use the system theme and firefox is on GTK according to google.
Help again?
org.gnome.desktop.interface color-scheme 'prefer-dark'
org.gnome.desktop.interface gtk-theme '(your-theme)-dark'
$ gsettings get org.gnome.desktop.interface color-scheme
No such key “color-scheme”
Btw, what version of Firefox are you using? Debian comes with ESR, so chances are, that firefox' "ui.systemUsesDarkTheme = 1" still works (it doesn't in newer releases).
So anyway do you think that if I upgrade to the latest Debian, your gtk command line will work?
I guess now I have a reason to upgrade.
Thank you!
What does that mean "gtk apps"? Is firefox a "gtk app"? Is vscode a "gtk app"? etc.