Tmpfs inode corruption: introducing inode64
chrisdown.name
chrisdown.name
A system that uses "our identifier space is approximately infinite" can be fragile in a surprising way. A system that uses "our identifier space is proportional to the physical size of the medium" is more humanly predictable in its limitations.
One of the earliest test versions of my patch actually did inode reuse using slabs just like you're suggesting, but there are a few practical issues:
1. Performance implications. We use tmpfs internally within the kernel in a lock-free manner as part of some latency-sensitive operations, and using slabs complicates that somewhat. The fact that we make use of tmpfs internally as the kernel makes this situation quite different than other filesystems.
2. Back when I was writing the patch, each memory cgroup had its own set of slabs, which greatly complicated being able to reuse inodes as slabs between different services (since they run in different memcgs).
After it became clear that slab recycling wouldn't work, I wrote a test patch that uses IDA instead, but I found that the performance implications were also not tenable. There are other alternative solutions but they increase code complexity/maintenance non-trivially and aren't really worth it.
A 64-bit per-superblock inode space resolves this issue without introducing any of these problems -- before you go through 2^64-1 inodes, you're going to have other practical constraints anyway, at least for the timebeing :-)
I've been through that hell before and it completely sucks.
BSD UFS will reuse an inode occasionally if you replace a file in a dir by delete + create (which makes sense, if you only cared about the efficiency of the dir structure).
But this meant that my caches which were keyed by device + inode + mtime, would fail to detect a file being updated.
As an obvious workaround, there was a 2 second delay into the system which seemed to fix things at first (apc.file_update_protection), but then it also stopped working for some people.
Because they were pushing data out to nodes through something like an rsync which also pushes an mtime update.
Anyway, the eventual workaround was to cache-break by ctime as well as mtime.
But it was hell to debug (months with 1 SEV1 ticket), because it would take a lot (a lot) to even triage the issue to the filesystem of why the file updates aren't going to the cache (& requests failing with API mismatches as the symptom).
Would not spring this on people without an explicit warning.
PS: Looks like ext4 does reuse them, whoa
Filesystem cache indexing should generally be done on the (name, type, inode, ctime) tuple (ideally include inode generation, but it's a fuss to get and doesn't make much difference in practice).
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
Thank you!
This is all from memory and I'm on my phone, so no citations. In any case, I don't think inode numbers were the problem.
If userspace wants to distinguish files, it normally does so with a (ino_t, dev_t) pair. With some nuance wrt inode generations if you aren't holding on to files yet want to guard against ino reuse, and some funkyness with overlayfs.
Most filesystems don't expose a way to access files by ino only, so the internal implementation generally doesn't key anything by them or rely on them itself; they're just a kind of of uniqueness cookie. But keying by them somewhere in the kernel implementation is feasible, if you scope it properly.
Also, the article does say this:
> On non-volatile filesystems, inode numbers are typically finite and often have some kind of implementation-defined semantic meaning (eg. implying where the inode is disk).
If you've got other means to know where the i-node is on disk, then why would a filesystem bother encoding that into the i-node numbering scheme?
Maybe the article is wrong here. I don't know. But if the question is where the confusion is coming from, the article definitely seems to be painting a picture where normal filesystems look up things based on i-node number and tmpfs is relatively unique in not doing so.
ext and XFS still largely work like this, though directories are now hashtables and I think XFS supports multiple independent arrays for inodes (or just for extent allocation? I don't remember). NTFS also looks a lot like a Unix file system with some weird growths on it, and not a whole lot like FAT.
A file handle just needs a pointer to where in memory the contents are stored, and the inode numbers are computed on the side because it's part of the interface.
You don't want to use the pointers as inode numbers directly, since for security reasons the kernel doesn't want to export valid kernel pointers to user space.
TFA makes this clear...