Linux NILFS file system: automatic continuous snapshots
dataswamp.org
dataswamp.org
The continuous snapshotting has saved me a couple of times; I’ve just mounted a version of the file system from few hours or weeks ago to access overwritten or deleted data. I use NILFS also on backup disks to provide combined deduplication and snapshots easily (just rsync & NILFS’ mkss, latter to make sure the “checkpoints” aren’t unnoticedly garbage collected in case the backup disk gets full).
Yes it's really sad, there we have a native and stable check-summing fs, and nearly no one knows about it.
Is it? Last I'd heard was
> nilfs2 store checksums for all data. However, at least the current implementation does not verify it when reading.
The snapshots are not automatic, but short of that it is pretty feature complete
Maybe on x86? I’ve tried repeatedly to use it on ARM for RaspberryPi where it would have been perfect, but always ran into various kernel panics as soon as the file system is mounted or accessed.
I’ve heard so many stories of SD card failures (against which snapshotting might be of no help) with RaspberryPi that I’ve decided to send any valuable data promptly to safety over a network. (Though, I personally haven’t had any problems with failing SD’s.)
A decade later and btrfs is still riddled with problems and incomplete, people are still using xfs and ext4 for lack of trust, one kernel dev has a side hobby trying to block openzfs, and excellent little projects like nilfs are largely unknown.
Can you elaborate?
That quote is from Ted T'so (https://en.wikipedia.org/wiki/Btrfs#History) or do you have a link to Linus' quote?
SF website of NILFS2 suggests that there are some unimplemented features, one of them being synchronous IO, which might have caused that issue?
https://nilfs.sourceforge.io/en/current_status.html
In some cases, the NILFS2 is safer storage for your data than ZFS. So NILFS might work for some simple usecases (eg. localy storing documents that you modify often), but it's certainly not ready to be deployed as generic filesystem. It's relatively slow and sometimes behaves bit weird. If something goes really bad, the recovery might be bit painfull. There is no fsck yet, nor community support. NILFS2 can self-heal itself to some extent.
I really like the idea of NILFS2 but at this point i would prefer patch adding continuous snapshotting to ZFS. Unlike NILFS2 the ZFS have lots of active developers and big community. While NILFS2 is almost dead. The fact it's been in kernel for quite some time and most people didn't even noticed it (despite it's very interresting features) speaks for itself.
Don't get me wrong. I wish that more developers get interested in NILFS2 and fix these issues and make it on par with EXT4, XFS and ZFS... But still ZFS has more features overall, so we might just add continuous snapshots in memoriam of NILFS2.
What cases? Do you just mean due to continuous snapshots protecting against accidental deletes or such, or are there more "under the covers" things it fixes?
https://www.reddit.com/r/btrfs/comments/fcntc6/qcow2_on_btrf...
You can either make a sub volume and mount it with nocow or set the +C file attribute to disable CoW:
http://blog.jim.nz/2015/12/04/btrfs-subvolume-with-nocow.htm...
Doesn’t really look like nilfs2 has this kind of flexibility, though
We now have storage systems (even SSDs) that are big enough to hold hundreds of millions of files. Those files can be a mix of small files, big files, temp files, personal files, and public files. Yet every file system must treat your precious thesis paper the same way it treats a huge cat video you downloaded off the Internet.
We need some kind of 'object store' where each object can be given a set of attributes that govern how the file system treats it. Backup, encryption, COW, checksums, and other operations should not be wasted on a bunch of data that no one really cares about.
I have been working on a kind of object file system that addresses this problem.
Couldn't one just just generate a filesystem per object if snapshots, etc., on a per object level is what one cared about? Wonder how quickly this would fall over?
> Backup, encryption, COW, checksums, and other operations should not be wasted on a bunch of data that no one really cares about.
This GP comment is a little goofy though. There was a user I once encountered who wanted ZFS, but a la carte. "I want the snapshots but I don't need COW." You have to explain, "You don't get the snapshots unless you have the COW", etc.
"When you enable nocow on your files, Btrfs cannot compute checksums, meaning the integrity against bitrot and other corruptions cannot be guaranteed (i.e. in nocow mode, Btrfs drops to similar data consistency guarantees as other popular filesystems, like ext4, XFS, ...). In RAID modes, Btrfs cannot determine which mirror has the good copy if there is corruption on one of them."[0]
[0]: https://wiki.tnonline.net/w/Blog/SQLite_Performance_on_Btrfs...
For instance, if you modify a single byte in a large file, you need to update the data on disk as well as the checksum in the block header, and other related data. Chances are, these are in different sectors, and also require re-reading in all the other data in the block to compute the checksum. Anywhere in that process is a chance for corruption of the original data and the update.
If the byte changes the final compressed size, it may not fit in the current block at all, causing an expensive (or impossible) re-allocation.
You could end up with the original data and update both invalid.
Writing out a new COW block is done all at once, and if it fails, the write failed atomically, with the original data still intact.
Not much different than any interrupted write though. And a COW needs to reread just as much.
> If the byte changes the final compressed size, it may not fit in the current block at all, causing an expensive (or impossible) re-allocation.
Something that you must always pay in a COW filesystem anyway? Is handled by other non-COW filesystems anyway.
Just because a filesystem isn't COW doesn't mean every change needs to be in place either. Of course, a filesystem that is primarily COW might not want to maintain compression for non-COW edge-cases and that is quite reasonable.
A COW write only needs to checksum the newly written bytes. A non-cow filesystem needs to checksum all data contained in the block (unchanged prior with now new values).
Additionally, a non-COW filesystem needs to update all metadata checksums/values for existing blocks. It's a much more pathological case (interrupted write wise) than a COW filesystem, because if it writes data, but hasn't written the checksum yet - the block is now corrupt. If it writes the checksum, but not the data, the block is now corrupt. And there is no way to know which one is correct post-facto, without storing the old data and the new data somewhere. Which has the exact same overhead or worse as COW. And the data in a FS block is usually many large multiples of the sector size, which makes writes pretty hard to do in any sort of atomic way. Journaling helps, but not with performance here! Since you'd need to store the prior values + the new values, or you're still guaranteed to lose data.
Compression wise, this isn't (as much) of an issue for COW filesystems, because it only needs to compress the newly written data, which can be allocated without concern to the previous allocation size, which is still there, allocated. It can mean less efficient compression if these are small, fragmented writes of course, which is why most of them have some sort of batching mechanism in place. Alternatively, it can copy a chunk of the block, though that can cause write amplification, and is usually minimized.
But you don't run across potential pathological fragmentation issues, like where you compress prior blocks which now take significantly less space, or new blocks take more space and require reshuffling everywhere.
No, a COW will re-read the entire block. Make the change and update the checksum and then write it back (to a new location, obviously). Way more than the newly written bytes - but way less than the entire file of course. Just as a non-COW fs will.
>If it writes the checksum, but not the data, the block is now corrupt. And there is no way to know which one is correct post-facto, without storing the old data and the new data somewhere.
Which is exactly what the journal is for - and you already have a journal. Or you don't update it in-place. Just because it isn't COW doesn't mean you always have to update in place.
>Compression wise, this isn't (as much) of an issue for COW filesystems, because it only needs to compress the newly written data, which can be allocated without concern to the previous allocation size, which is still there, allocated. It can mean less efficient compression if these are small, fragmented writes of course, which is why most of them have some sort of batching mechanism in place. Alternatively, it can copy a chunk of the block, though that can cause write amplification, and is usually minimized.
You can do exactly the same for a non-COW filesystem.
>But you don't run across potential pathological fragmentation issues, like where you compress prior blocks which now take significantly less space, or new blocks take more space and require reshuffling everywhere.
COW has the same issue. COW always need to "reshuffle"(?) data somewhere.
They don't work the way you are asserting.
The failure modes are well documented - and got worse in many cases trying to work around performance issues due to journaling, but the journaling doesn’t resolve the issue fully because they can’t store all the data they need without making the performance issues worse. See https://en.m.wikipedia.org/wiki/Ext4 and ‘Delayed allocation and data loss’ for one example.
This isn’t a solved (or likely solvable in a reasonable way) problem with non-COW filesystems, which is one of the reasons why all newer filesystems are COW. The other being latency hits from tracking down COW delta blocks aren’t a big issue now due to SSDs and having enough RAM to have decent caches and pre-allocation buffers.
Also, COW doesn’t need to allocate (or re-read/re-checksum) the entire prior block when someone changes something, unlike modify in place. Due to alignment issues, doing SOME usually makes sense, but it’s highly configurable.
It only needs to add new metadata with updated mapping information for the updated range in the file, and then checksum/write out the newly updated data (only), plus or minus alignment issues or whatever. It acts like a patch. That’s literally the whole point of COW filesystems.
Update in place has an already allocated block it has to deal with in real time, either now consuming less space in it’s already allocated area (leaving tiny fragmentation) or by having to allocate a new block and toss the old one, which will have worse real time performance than a COW system, as it’s doing the new block allocation (which is more space than a COW write, unless the COW write is for the entire blocks contents!), plus going back and removing the old block.
ZFS record size for instance is just the maximum size of one of the patch ‘blocks’. The actual records are only the size of the actual write data + Filesystem overhead.
ZFS only then goes back and removes old records when they aren’t referenced by anyone, which is typically async/background, and doesn’t need to happen as part of the write itself.
This allows freeing up entire regions of pool space easier, and fragmentation becomes much less of an issue.
That is just a matter of priorities then. And just because you might opt to not update in place in some situations doesn't mean that you can never do it.
I'm not sure what you mean by "Delayed allocation and data loss", I don't find it relevant to this discussion at all since that isn't about filesystem-corruption but application data corruption. And COW also suffers from this - unless you have NILFS/automatic continuous snapshots. Now with COW you probably have a much greater chance of recovering the data with forensic tools (also discussed in this thread regarding ZFS) but with huge downsides and hardly an relevant argument for COW in the vast majority of usecases anyway.
ZFS minimum block size corresponds to disk sector size so for most practical purposes it is the same as your typical non-COW filesystem there. Writing 1 byte requires you to read 4 kb, update it in memory, recalculate checksum, and then writing it down again.
How you remove old records shouldn't depend on COW should it?
My only statement was that checksums isn't in any way dependent on COW.
The discussion about compression is invalid as it is a common feature of non-COW filesystems anyway.
Haven't seen a proper argument for the corruption claims. And that you get corrupted data if you interrupt a write is not a huge deal. Mind you corrupted write. Not corrupted filesystem. The data was toast anyway. A typical COW would at best save you one "block" of data which is hardly worth celebrating anyway. Your application will not care if you wrote 557 out of 1000 blocks or 556 out of 1000 blocks your document is trashed anyway. You need to restore from backup (or from a previous snapshot, which of course is typical killer feature of COW)).
There are also several ways to solve the corruption issue. ReFS for instance has data checksums and metadata checksums but only do copy-on-write for the metadata. (edit: was wrong about this, it uses COW for data too if data checksumming is enabled)
dm-integrity can be used at a layer below the filesystem and solves it with the journal https://www.kernel.org/doc/html/latest/admin-guide/device-ma...
Yes, COW is popular and for good reasons. As is checksumming. It isn't surprising that modern filesystems employ both. Especially since the costs of both have been becoming less and less relevant at the same time.
I suppose I don't understand this. Why would this be the case?
Oh look a hobby-lawyer!! Please Linus, license your code "correctly" it's called ISC not GPL.
The challenge with your thesis here is that the only one who can know what is ‘that important’ is YOU, and your decision making and communication bandwidth is already the limiting factor.
For many users, that cat video would be heartbreaking to lose, and they don’t have term papers to worry about.
So having to decide or think what is or is not ‘important enough’ to you, and communicate that to the system, just makes everything slower than putting everything on a system good enough to protect the most sensitive and high value data you have.
Getting each piece of data properly classified will always be a challenge (AI or other tools may help with that), but it would still be nice to be able to do it. If I have a 50GB video file that I could easily re-download off the Internet, it would be nice to be able to turn off any security, redundancy, or integrity features for it.
I wonder how many petabytes of storage space is being wasted by having multiple backups of all the operating system files that could be easily downloaded from multiple websites. Do I really need to encrypt that GB file that 10 million people also have a copy of? Am I worried if a single pixel in that high resolution photo has changed due to bit rot?
Indeed you don't. Poettering has a similar idea in [1] (scroll down to "Summary of Resources and their Protections" for the tl;dr table), where he imagines OS files are only protected by dm-verity (for Silverblue-style immutable distros) / dm-integrity (for regular mutable distros).
[1]: https://0pointer.net/blog/authenticated-boot-and-disk-encryp...
The big issue regarding duplication is really more a identification/‘supply chain’ issue. The last thing I want to be doing is trying to figure out how to get that other file from somewhere (that works), from whoever ‘has another copy’, when it gets mangled and I need a replacement ASAP.
If you’re thinking of our local filesystems as potentially just a cache, we have no easy or secure way right now to fingerprint or recreate the other entries in the cache from sources (minus browser caches or the like, but even then, re-retrieving it may return different or not content).
So backups are really keeping the computing equivalent of local ‘ability to manufacture’ handy as a mitigation against risk. It’s not waste, anymore than keeping the ability to manufacture it’s own tanks and weapons in-country is a waste for a nation. It’s an insurance cost against real world problems, and causes a moral hazard with other actors if not done.
And CRC32/CRC32C (even in Java!) is > 16GB/s per core in modern processors, and more than adequate for block sized (typ. < 32MB) checksums. Blake3 is > 2GB/s per core, and is more than adequate for… well every use case we’re currently aware of.
Modern OS’s really don’t have any excuses for not doing it.
Depends on where that cat video is / how it ended up on the disk.
The user explicitly saved it to their user-profile Downloads directory? Yeah, sure, the user might care a lot about preserving that data. There's intent there.
The user's web browser implicitly saved it into the browser's cache directory? No, the user absolutely doesn't care. That directory is a pure transparent optimization over just loading the resource from the URL again; and the browser makes no guarantees of anything in it surviving for even a few minutes. The user doesn't even know they have the data; only the browser does. As such, the browser should be able to tell the filesystem that this data is discardable cache data, and the filesystem should be able to apply different storage policies based on that.
This is already true of managed cache/spool/tmp directories vis-a-vis higher-level components of the OS. macOS, for example, knows that stuff that's under ~/Library/Caches can be purged when disk space is tight, so it counts it as "reclaimable space"; and in some cases (caches that use CoreData) the OS can even garbage-collect them itself.
So, why not also avoid making these files a part of backups? Why not avoid checksumming them? Etc.
They can be ways to restore to a point in time deterministically - but then they are absolutely needed to do so! Otherwise, the software is going to be acting differently with a bunch of data gone from underneath it, no?
Check summing is more about being able to detect errors (and deterministically know if data corruption is occurring). So yes, absolutely temporary and cache files should be checksummed. If that data is corrupted, it will cause crashes of the software using them and downstream corruption after all.
Why would I not want that to get caught before my software crashes or my output document (for instance) is being silently corrupted because one of the temporary files used when editing it got corrupted to/from disk?
...no? I don't care if a video in my browser's cache ends up with a few corrupt blocks when I play it again a year later. Video codecs are designed to be tolerant of that. You'll get a glitchy section in a few frames, and then hit the next keyframe and everything will clean up.
In fact, most encodings — of images, audio, even text — are designed to be self-synchronizing in the face of corruption.
I think you're thinking specifically of working-state files, which usually need to be perfect and guaranteed-trusted, because they're in normalized low-redundancy forms and are also used to derive other data from.
But when I say "caching", I'm talking about cached final-form assets intended for direct human consumption. These get corrupted all the time, from network errors during download, disk storage errors on NASes, etc; and people mostly just don't care. For video, they just watch past it. For a web page, they hard-refresh it and everything's fine the second time around.
If you think it's impossible to differentiate these two cases: well, that's because we don't explicitly ask developers to differentiate them. There could be separate ~/Library/ViewCache and ~/Library/StateCache directories.
And before you ask, a good example of a large "ViewCache" asset that's not browser-related: a video-editor render-preview video file (the low-quality / thumbnail-sized kind, used for scrubbing.)
Because merely trying to read it can cause severe issues at the filesystem level.
I take it you haven’t dealt with failing storage much before?
> Especially when it’s the filesystem metadata which gets corrupted.
We're not talking about filesystem metadata, though. Filesystem metadata is all "of a piece" — if you have a checksumming filesystem, then you can't not checksum some of the filesystem metadata, because all the metadata lives in (the moral equivalent of) a single database file the filesystem maintains, and that database gets checksummed. It's all one data structure, where the checksumming is a thing you do to that data structure, not to individual nodes within it. (For a tree filesystem like btrfs, this would be the non-cryptographic equivalent of a merkle-tree hash.) The only way you could even potentially turn off filesystem features for some metadata (dirent, freelist, etc) nodes but not others, would be to split your filesystem into multiple filesystems.
No, to be clear, we're specifically talking about what happens inside the filesystem's extents. Those can experience corruption without that causing any undue issues, besides "the data you get from fread(3) is wrong." Unlike filesystem metadata, which is all required for the filesystem's integrity, a checksumming filesystem can choose whether to "look" inside file extents, or to treat them as opaque. And it can (in theory) make that choice per file, if it likes. From the FS's perspective, an extent is just a range of reserved disk blocks.
Now, an assumption: only storage arrays use spinning rust for anything any more. The only disk problems consumer devices face any more are SSD degradation problems, not HDD degradation problems.
(Even if you don't agree with this assumption by itself, it's much more clear-cut if you consider only devices operated by people willing to choose to use a filesystem that's not the default one for their OS.)
This assumption neatly cleaves the problem-space in two:
- How should a filesystem on a RAID array, set up for a business or prosumer use-case, deal with HDD faults?
- How should a single-device filesystem used in a consumer use-case deal with SDD faults?
The HDD-faults case comes down to: filesystem-level storage pool management with filesystem-driven redundant reads, with kernel blocking-read timeouts to avoid hangs, with async bad-sector remapping for timed out reads. Y'know: ZFS.
While the SDD-faults case comes down to: read the bad data. Deal with the bad data. You won't get any hangs, until the day the whole thing just stops working. The worst you'll get is bit-rot. And even then, it's rare, because NAND controllers use internal space for error-correction, entirely invisibly to the kernel. (See also: http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/)
In fact, in my own personal experience, the most likely cause of incorrect or corrupt data ending up on an SSD/NVMe disk, is that the CPU or memory of the system is bad, and so one or the other is corrupting the memory that will be written to disk before or during the write. (I've personally had this happen at least twice. What to look for to diagnose this: PCIe "link training" errors.)
And one caused a kernel panic because the fs metadata now pointed towards impossible values.
So tell me again how none of this matters?
And doing that is not at all easy. For all anybody knows your cat video is “worth more” to you than your thesis paper. How can you get the system to determine the worth of each file without manually setting an attribute each time you create a file? And if you let the system guess, the cost of failure could be very high! What if it decided your thesis paper was worthless and stored it will a lower “integrity” (or whatever you call the metric)?
I dunno. Storage is getting cheaper all the time and it might just be easier to fuck it and treat all files with the same high level of integrity. Maybe it would be so much work for a user to manually manage they’d just mark everything the same?
If I was copying an important file into an unprotected environment, I could change how it was handled (likewise if I was downloading some huge video I didn't care about into a system where the default protection was set to high).
I agree that if you have 100 million files, then it could be nearly impossible to classify every single one of them correctly.
I thought ZFS had some sort of unique settings like this.
I think directories or volumes having different properties and you having it split up as /consumer-media /work-media /work /docs /credentials etc may be the way to go.
Then you can set integrity, encryption etc separately, either at filesystem level or as part of the software-level backup strategy.
NILFS is, by default, a filesystem that only ever appends until you garbage collect the tail. It doesn't really "snapshot" in the way that ZFS or btrfs do, because you can just walk the entire history of the filesystem until you run out of history. The snapshots are just bookmarks of a consistent state.
We combined nilfs + our process snapshotting tech (we tried to mainline it, but it didn't go, but many of the concepts ended up in CRIU though) + our remote display + screen reading tech (i.e. normal APIs) to create an environment that could record everything you ever saw visually and textually. enable you to search it and enable you to recreate the state as it was at that time with non noticeable interruption to the user (processes downtime was like 0.02s).
i.e. combining criu with nilfs (but a lot of the work that we did to get process downtime to minimal numbers requires being in kernel, as described in paper) and unsure criu can do it.
In addition our screenrecording mechanism was our own "proprietary" (not really proprietary as fully described in research papers, but also not a standard) and something that was built as an X display driver 15 years ago (so not directly usable today even if code is available). Could probably duplicate it with vnc based screencasting. vnc didn't work for us as we needed better performance (i.e. it was built to demonstrate remote display of video and games and there was no real remote audio setup back then so we had to create our own).
the "text" search just used gnome's accessible API much like a screenreader would do (with a bit of per application optimizations as can filter out things like menus and the like, primarily was to dump text out of terminals, firefox and perhaps open office and maybe even a pdf reader if memory serves me correctly, but a long time ago).
1. being the layer upon which AI-assisted computing is trained and built
2. being used for video game QA. For certain heisenbugs it'd be invaluable to be able to restore point-in-time repro to resume just before it happened
https://www.cs.columbia.edu/~nieh/pubs/sigmetrics2010_scribe... https://www.cs.columbia.edu/~nieh/pubs/sigmetrics2011_transp...
(Not a "gotcha" question, a legitimate question.)
It's also worth knowing NILFS2 has checkpoints and snapshots. What you actually get are continuous "checkpoints". These can be upgraded to snapshots at any time with a simple command. Checkpoints are garbage collected, snapshots are not (until they are downgraded back into checkpoints).
This was made more frustrating by the lack of any tooling such as fsck to help me diagnose the issue. The only reason I figured out it was a bug was that I booted a live CD to try to rescue the system and it booted fine.
When I finally replaced that laptop I went back to ZFS and scripted snapshots. As much as I want to, I just can't recommend NILFS for daily use.
Good guess, but no:
https://github.com/torvalds/linux/tree/master/fs/nilfs2
> How did Linus not go on a rampage after breaking userspace for an entire year?
I would very much like to know that as well. Any chance it didn't get reported (at least, not as "this broke booting")?
Edit: Looking through the archives, it looks like my memory was somewhat uncharitable. It was reported in November and directly patched in June (https://marc.info/?l=linux-nilfs&m=159154670627428&w=2) so about 7 months after reporting. Not sure what kernel release that would've landed in, so could've been closer to 8.
It is a bug that prevents the kernel from booting. That's bad, but that's not the same thing. That's not a userspace compatibility issue such as the ones Linus chases. The user space isn't even involved if the kernel cannot boot. Or if it is actually a userspace program that causes a kernel crash, it is a crash, which is not really the same thing as an API change (one could argue, but that's a bit far-fetched, the intents are not the same, etc - I don't see Linus explode on somebody who introduced a crash the way he would explode on someone changing a userspace API).
Linus' commandment about not breaking userspace is frequently misunderstood. He wants to ensure that user-space /programs/ do not break (even if they rely on buggy behavior that made it into a release), not that the /user/ will never see any breakage of the system whatsoever, which is of course an impossible goal. Device drivers and filesystems are firmly system-level stuff, bugs and backwards-incompatible changes in those areas are regrettable but happen all the same.
I had troubles with un-popular file systems as root file system when the initrd was not built properly. So sysresccd is always good to have in reach.. Saying this, I think I won't have any other file system on root besides the default of the distro. Data which require special care are on other partitions.
It does without a performance penalty compared to other logging filesystems.
And without using additional space forever. The backlog rotates forward continuously.
It's a really unique feature that makes a lot of sense for desktop use, where you might want to recover files that were created and deleted after a short time.
yeah. it's already so terribly slow that it's unlikely that taking snapshots can make it any slower :-D
I don't remember what benchmark I ran before deciding to run it on my laptop. Given my work at the time probably pgbench, but I couldn't say for sure. It was long enough ago I also might've been benchmarking against ext3, not 4.
What is the basis for comparison? Sounds like a pretty meaningless statement at its face.
Of course if you haver server handling let's say video files things will be very different. And there are some users who process huge amounts of data.
I run 2 lvm snapshots (daily and weekly) on my home partition for years. Write performance is abysmal if you measure it, but you don't note it in daily development work.
Sounds like it could serve as a basis for a Linux implementation of something like Apple Time Machine.
So you can already do that with zfs: take a snapshot and send it to the backup drive.
ZFS and btrfs users are already living in the future:
inotifywait -r -m --format %w%f -e close_write "/srv/downloads/" | while read -r line; do
# command below will snapshot the dataset
# upon which the closed file is located
sudo httm --snap "$line"
done
But i still can imagine failure modes, eg. inotify might start acting weird when ZFS remounts the watched directory, OOM killer terminates it without anyone noticing, bash loop go haywire when package manager updates that script (bash is running directly from the file and when it changes during execution, it might just continue running from the same byt offset in completely different script).
All these things actualy happened to me in the past. Not to say that if you have multiple datasets in ZFS you cannot inotify wait on all of them at once, so you will have to manage one bash process per dataset. And performance of bash and sudo might not be that awesome.
So for real reliability you would probably want this to actualy run in ZFS/kernel context...
See: https://github.com/kimono-koans/httm
> But i still can imagine failure modes, eg. inotify might start acting weird when ZFS remounts the watched directory, OOM killer terminates it without anyone noticing, bash loop go haywire when package manager updates that script (bash is running directly from the file and when it changes during execution, it might just continue running from the same byt offset in completely different script).
I mean, sure, scripts gonna script. You're gonna have to make the POC work for you. But, for instance, I'm not sure half of your issues are problems with a systemd service. I'm not sure one is a problem with a well designed script, which accounts for your particular issues, and a systemd service.
> All these things actualy happened to me in the past. Not to say that if you have multiple datasets in ZFS you cannot inotify wait on all of them at once, so you will have to manage one bash process per dataset. And performance of bash and sudo might not be that awesome.
Yes, you can?
Just re this POC, you can inotifywait a single directory, which contains multiple datasets, and httm will correctly determine and snapshot the correct one upon command. Your real bottleneck here is not sudo or bash. It's the zfs command waiting for a transaction group sync, or arranging for the trans group (or even something else, but its definitely zfs?), to snap.
You can also use `httm -m` to simply identify the dataset and use a channel program and/or a separate script to sync. sudo and bash may not have the performance for your use case, hell, they are composable with everything else?
> So for real reliability you would probably want this to actualy run in ZFS/kernel context...
Yeesh, I'm not sure? Maybe for your/a few specific use cases? Note, inotify (a kernel facility) is your other bottleneck. You're never going to want to watch more than a few/10s of thousand files. The overhead is just going to be too great.
But for most use cases (your documents folder)? Give httm and inotifywait a shot.
What if I compare it to BTRFS + Snapper? No performance penalty there, plus checksumming.
Even background snapper cleanups will start to take a toll, since even if they are done with ionice they tend to block simultaneous accesses to the filesystem while they are in progress. If you have your root on the same filesystem, it's not pretty -- lots of periodic system-wide freezes with the HDD LEDs non-stop blinking. I tend to limit snapshots always to < 20 for that reason (and so does the default snapperd config).
There are some solutions like photorec (which now has ZFS support), but it expects you can identify the file by footprint of its contents, which was not the case. Also many of these solutions would require ZFS to go offline for forensic analysis and that was also not possible because lots of other clients were using the same pool at the time.
So this had failed me and i really wished at the time that ZFS had continuous snapshots.
BTW on ZFS i use ZnapZend. It's second best thing after continuous snapshots:
https://www.znapzend.org/ https://github.com/oetiker/znapzend/
There are also some ZFS snapshotting daemons in Debian, but this is much more elegant and flexible.
But since znapzend is userspace daemon (as are all ZFS snapshoters) you need some kind of monitoring and warning mechanism for cases something goes wrong and it can't longer create snapshots (crashes, gets killed by OOM or something...). In NILFS2 every write/delete is snapshot, so you are basicaly guaranteed by kernel to have everything snapshoted without having to watch it.
I cannot tell if NILFS is doing this too, with zfsnap I maintain different retention times. 5-minutely for 1hour, hourly for 1day, daily for a week. That are less than 60 snapshots. The older ones are cleaned up.
In addition, zfs brings compression and encryption. That's why I have it on the laptops, too.
At this point, the view seems to be "attributes happen in the file system, indexing happens in user space".
Especially on linux.
Part of the reason is, as i understand it, the surface/complexity of including query languages in the kernel, which is not horribly unreasonable
So all the common FSen have reasonable xattr support, and inotify/etc that support notification of attribute changes.
The expectation seems to be that the fact that inotify might drop events now and then is not a dealbreaker. The modern queue length is usually 16384 anyway.
I'm not saying there aren't tradeoffs here, but this seems to be the direction taken overall.
I actually would love to have an FS with native indexed xattr and a way to get at them.
I just don't think we'll get back there again anytime soon.
Really what I'd like is just to search for some specific tags, or maybe list a directory excluding some tag, or similar. For bonus points, maybe a virtual directory that represents a search like this, and which "contains" the results of that search. (A "Search Folder")
I'll check out BeFS. Thanks!
I would very much welcome a filesystem that breaks away from the directories/files paradigm. Any time-based data store would greatly benefit from that.
My point was more on the tracks of a filesystem where a single file can be overwritten over and over again, and it's up to the filesystem to transparently ensure the full capacity of the disk is put towards retaining old versions of the file.
https://docs.ceph.com/en/latest/rados/configuration/storage-...
BUT
if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, the only reason they might not is due to "security through obscurity".
now, what I have argued is that an append only file system which works in a SAN like environment (i.e. you have random reads, but only append writes properties that are enforced remotely) could give you that, but to an extent you'd still get a similar behavior by just exporting ZFS shares (or even as block devices) and snapshotting them regularly on the remote end.
How so?
Let's say you have one machine holding the actual data for working on it. And some backup server. You could use btrfs send over ssh and regularly btrfs receive the data on the backup machine. Even it they got encrypted by ransomware they wouldn't be lost in the backups. As long as they're not deleted there how could a compromised work machine compromise the data on the backup machine?
That btrfs receive may well be exploitable (because of some bug) if you feed it with malicious data is IMO a different topic.
that would fit with my description (at least in a sense) it enables an "append" only mode to a remote side (where "append" means cant mess with previous snapshots remotely).
https://docs.oracle.com/cd/E37670_01/E37355/html/ol_srbackup...
while it doesn't allow you to delete a subvolume (i.e. what a snapshot is built off of), it does seem like one could conceptually just overwrite it via send/recieve? perhaps its a bit smarter and doesn't let snapshots be modified, but I'm now a bit unconvinced that it can be viewed as simply an "append only" interface.
Don't think so.
The man page for btrfs receive states:
> btrfs receive will fail in the following cases:
> 1. receiving subvolume already exists
I also just tried it out. I created some subvol1-ro in a source filesystem, then sent it to the destination filesystem. Then deleted the source subvol1-ro. Then created a new subvol1-ro containing a different file and tried to send it to the same destination.
Result:
> ERROR: creating subvolume subvol1-ro failed: File exists
A received ro snapshot can be used as a base for a new writable snapshot.
It didn't do directories so was certainly not as good as snapshot but we're talking 40 years ago!
>> DEC attempted to replace it with a log-structured file system file system named Spiralog first released in 1995. However, Spiralog was discontinued due to a variety of problems, including issues with handling full volumes.
Every file has a version number, which defaults to 1 if no other versions of the same filename are present (otherwise one higher than the greatest version). Every time a file is saved, rather than overwriting the existing version, a new file with the same name but an incremented version number is created. Old versions can be deleted explicitly, with the DELETE or the PURGE command, or optionally, older versions of a file can be deleted automatically when the file's version limit is reached (set by SET FILE/VERSION_LIMIT). Old versions are thus not overwritten, but are kept on disk and may be retrieved at any time. The architectural limit on version numbers is 32767. The versioning behavior is easily overridden if it is unwanted. In particular, files which are directly updated, such as databases, do not create new versions unless explicitly programmed.
> A checkpoint is a snapshot of your system at a given point in time, but it can be deleted automatically if some disk space must be reclaimed. A checkpoint can be transformed into a snapshot that will never be removed.