Hacker News Reader

Comment by adrian_b | Hacker News Reader

Beware that this script only uses rsync with the "--archive" flag.

This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises.

rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies.

Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most file metadata and they usually need many non-default flags to make complete file copies.

Nevertheless, rsync is the best of all alternatives, because when invoked correctly it accurately copies all file metadata even between different file systems and different operating systems, in cases when other copying utilities may lose some file metadata without giving any errors or warnings for the users.

View on HN

Better than rsync is using support for snapshots and replication in the filesystem, such as described for ZFS in

https://serverfault.com/questions/842531/how-to-perform-incr...

Btrfs and apfs also support auto-snapshotting and replication, but I don't have experience with them.

ZFS snapshots also have the added benefit of not taking up duplicate space. And, should a Linux ransomware ever come into existence, snapshots help against them.

Btrfs and apfs also support copy-on-write snapshots.

Also, snapshots are NOT backups. And one definitely should not rely on snapshots along to keep their files safe.

They aren’t; a backup should preferably be offsite, but nothing is perfect

Rsync also doesn't track hardlinks by default! I use "-avPzHAXS" for my backups.

Also have a look at --numeric-ids, otherwise it translates UIDs back and forth according to real user names - which will end up in a terrible mess, especially when you are restoring the backup from a live "CD" (which has different UID ←→ username mapping than the target system).

I use getfacl to save permissions correctly, in case the backup server uses different user/group mappings. And setfacl after restoring.

(I wish this was handled by rsync.)

This also presumes that the target's rsync implementation and filesystem supports the same metadata.

Most modern file systems have been updated to support equivalent metadata.

However many old versions of file systems lacked support for things like high-resolution timestamps, extended attributes or access-control lists. Because such support has been introduced much later, there are a lot of programs for archiving, backup and copying which lose such metadata, at least with their default options.

While equivalent metadata exists on Linux XFS and EXT4, Windows NTFS, FreeBSD UFS and ZFS etc. each platform has different APIs for accessing file metadata (and on Linux the API may vary even between the many available file systems), so only few programs, like rsync, are able to copy everything while taking care to use the correct API to replicate the metadata without losing information.

Hey Adrian, thanks for the improvements and hints!

For years I had a custom script sync my ~/.ssh directory on my primary workstation to my laptop, to pick up new keys and config changes. It failed after I switched from Ubuntu to Fedora, and I was surprised to discover --xattrs fixed it.

tl;dr try this if rsync fails in unexpected ways:

  echo "Syncing ~/.ssh directory"
  rsync --archive --delete --xattrs ~/.ssh/ laptop:.ssh/

Probably because 99% of users do not have or need xattrs/acls.

Reply on news.ycombinator.com