Understanding Disk Usage in Linux
ownyourbits.com
ownyourbits.com
The article is well worth reading as it's reasonably comprehensive, including a small foray into COW filesystem.
(Knowing such things makes me feel kind of old).
Furthermore, is it possible to say how much "space" you would use if you were to create a file with a given size, accounting for block-size, fragmentation, and metadata? Matters such as block-size, inode usage, and metadata seem to make this very difficult even if you add special integration to the userspace application, for example by using stat or statfs. This could help prevent quota overruns for example.
These seem like hard problems unfortunately, and I suspect the best solution is to just create separate disk partitions for each quota group.
Secondly: disk usage accounting for metadata as well as regular file data may or may not be tricky. ZFS always tells you how much data+metadata is used by a file, helped by metadata itself being dynamically allocated on ZFS like everything else is. File systems like ext4 that have fixed metadata locations on disk don't report back metadata allocation with the file; it wouldn't really be useful to see this information since removing the file doesn't free any metadata in the ext4 case.
I suppose project quotas as outlined here would allow multi-group support though.
Another option could be sparsely-provisioned COW LVM volumes.
Quoting "man":
"When --reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails"
Q1: this is copy-on-write, right?
Q2: once/if the command completes successfully, are there any (potential) dangers (e.g. if I then immediately delete the original file) or can the new file be treated 100% as if I had copied it the classical way (without that option)?
Thx
2) It's intended to be treated 100% as if it had been copied the classical way. It's not a hardlink or a symlink and can be treated as a completely new file
Any recommendations about a utility (working on a layer higher than the filesystem) that can synchronize only the chunks of a specific file that has changed since the last sync and which generates as final result a "normal" file (no behind-the-curtains database - just copy the identical data from the local/old file and the changes/new data from the remote location to get a normal up-to-date local file)?
I think that when I tested this with rsync a long time back I did a similar mistake like this one https://stackoverflow.com/questions/28819379/block-level-cop... (doing a sync not over a network) and/or this one https://www.reddit.com/r/linux/comments/3nhx0p/rsync_block_l... (doing a sync by adding/removing some bytes instead of just changing their values, as it would happen in the preallocated img-file of the VMs).
I'll test it again - thanks a lot for the info!!
If either file is written to, it's COW.
You can treat both files as if they are normal files, as if they're identical copies. It's just that the duplication doesn't take up additional space other than metadata.
A possible caution, just in general, on Btrfs is that each subvolume gets its own stash of inodes. In a subvolume, inode numbers are unique, you won't see the same inum in use more than once. But inode numbers aren't unique on the volume. So two completely unrelated files can have the same inum but they would be in different subvolumes (including snapshots where this is particularly noticable because a snapshot is really just a pointer to the parent subvolume and all the inums are unchanged.)