How difficult is it to preserve metadata when moving a file? | Hacker News Reader

How difficult is it to preserve metadata when moving a file?

yakking.branchable.com

48 pointsashitlerferad13 comments

This is part 4 of a series of articles about writing a custom "mv" command. Here's the first one: http://yakking.branchable.com/posts/moving-files-1-copying/

Here's the whole list: http://yakking.branchable.com/tags/move/

I've had a hard time finding a low-end backup solution (i.e., for personal use) that retains and restores all metadata.

Long ago, I assumed that backup (and move) applications are relatively simple. But it has never seemed to work out that way.

Traditionally tar(1) was used as it preserves most metadata. Modern versions even include optional support for ACLs and extended atributes (xattr). It used to be common to see command lines and scripts that piped "tar c" into "tar x" directly as a way of copying files while utilizing tar's metadata-preserving features.

Today, back-to-back tar is rarely needed, as cp itself already has options to preserve metadata. In particular, "-a","--archive" turns all the normal features (including --recursive), making metadata-preserving backups as easy as:

    cp -a $SRC $DST

which is a shorthand for

    cp --recursive --no-dereference \
       --preserve=context,links,mode,ownership,timestamps,xattr

Just the "-a" option is enough for most simple local backups. Fancier backups are possible with cp's "--backup=numbered" and --reflink=auto features, but my favorite just-works solution for backups is rdiff-backup[1]. It preserves everything, uses rsync internally for minimal data transfer on incremental backups, and many more features.

[1] http://www.nongnu.org/rdiff-backup/

I still use tar, for example to copy an entire Linux distribution to a new drive is just:

    mount /dev/sdb1 /m

    tar cf - . --exclude=./m --exclude=./proc --exclude=./sys | (cd /m; umask 0; tar -xf -)

    mkdir /m/proc /m/sys

    grub-install --root-directory=/m /dev/sdb

(except you must also adjust UUIDs in /etc/fstab)

I'm a big fan of fsarchiver (https://www.fsarchiver.org/). I've successfully used it for restoring full Windows 7 systems (NTFS), which implicitly requires getting all the metadata correct (e.g., alternate data streams).

Thanks; it looks great on first impression. I'll give it a try.

Don't forget MS NTFS files support multiple data streams [1] and OS X supports resource forks [2]. I dislike both of these partly because they don't fit within the set of posix commands and command options I'm familiar with.

[1] https://technet.microsoft.com/en-us/library/cc961812.aspx

[2] https://en.wikipedia.org/wiki/Resource_fork

A discussion of extended attributes, ACLs and security labels (if you want those too) appear to be missing?

These things are a deep dark pit of problems and it isn't entirely unreasonable to just give up on them, but every once in awhile someone tries to do something useful with them so it's at least important to be aware that they exist.

I believe Richard has those planned for another article, this is a series still in development :-)

Ahh. :)

Apparently extremely difficult to impossible since the date created on my photos never seems to stick.

Use 'jhead' and sync file timestamp to exif data:

http://www.sentex.net/~mwandel/jhead/

Hopefully at some point you will mention the fact that not everyone has the same idea about what a file is (streams, extended attributes) or what metadata is (ACLs, versions, other timestamps)...

Reply on news.ycombinator.com