Comment by grepular | Hacker News Reader

Comment by grepular | Hacker News Reader

I started using Restic recently. It's good and I'm going to continue using it. That said, there are a couple of bad problems with it:

Firstly, if you want to prune old backups, e.g keep the last N1 hourly backups, and the last N2 weekly backups, etc, then it has that ability, however whilst it's doing it, the client has to download and upload a tonne of data in order to repackage the backup files that contain some data that needs removing and other data which doesn't.

Secondly, I've set up an "append only" system, where my various hosts can append to their own backups, but not overwrite or delete them. I wanted the backup server to be unable to read the backups (easy enough, don't supply the encryption keys to the backup server), however at the same time I wanted the backup server to be able to automatically prune old backups. It can not do that without the key. I don't want to give it the keys to the backups as then a compromised backup server means all of my hosts data are suddenly compromised.

How would you expect the backup server to be able to prune old encrypted backups if it can't decrypt them? How should it know what to delete?

The backup server should have information about when a particular backup was made, it should be able to prune based on backup date without having the ability to read any of the data inside the backup.

I assume the backups are incremental, so you cannot just toss old encrypted files without losing backed up files.

Worse, it's deduplicated, using a similar algorithm to git. You can forget old backups and theoretically that could be done without decrypting but deleting the actual data will require a decrypt.

This doesn't make any sense when the backups are deduplicated (or incremental). A given file/block/whatever unit of data isn't uniquely associated with a single backup.

I'm no encryption expert, but depending on how the data is partitioned it can be easier to securely prune encrypted data than unencrypted data. For example, securely wiping an encrypted drive is as easy as tossing the key.

If you wish to "securely" wipe an encrypted drive, it's best to write over the old data a few times using randomly generated keys, tossing the keys each time you do a write over.

That sounds pointless. If you have real crypto, tossing the key (in a way that it cannot be recovered) is absolutely enough.

It's not pointless - it's pretty standard practice.

For example, although drives in Google Cloud are encrypted at rest, once they are decommissioned the drives are physically destroyed: https://cloud.google.com/security/deletion/#ensuring_safe_an...

> When a hard drive is retired, authorized individuals verify that the disk is erased by overwriting the drive with zeros and performing a multi-step verification process to ensure the drive contains no data.

Thanks. I guess the reason is to avoid future decryption due to either a new crypto attack or to avoid being vulnerable to potential errors in key generation.

This is massive overkill. Once is enough. If the drive is already encrypted, just wipe the key.

Perhaps by using homomorphic encryption?

https://en.wikipedia.org/wiki/Homomorphic_encryption

ZFS does that with snapshots already. (ZFS is not encrypted, but the principle is the same)

I believe immutable trees would be the correct algorithm here.

The algorithm relies on a tree structure, where the leaves are the files. Every time a new snapshot is written, every leaf node changed writes a new path to the root node, while leaving the rest of the tree intact.

When a snapshot needs to be deleted, you just remove all the old root nodes and have a garbage-collection like-process delete all the old files.

EDIT: Look at example here: https://en.wikipedia.org/wiki/Persistent_data_structure#Tree... . When xs is deleted, nodes a,c,f can also be deleted.

EDIT2: I feel like I can never make posts like these while keeping it short and sweet, and adding enough details. ZFS can be encrypted, but accessing any files is right at its fingertips, not over a slow network connection. And there are caveats about how to encrypt the metadata, but you do not need to decrypt the whole backup to figure out what to do delete.

AFAIK that is exactly how restic works. You use `restic forget` to remove the roots (snapshots), then `restic prune` garbage collects unreachable blobs.

ETA: However, nearly all data in restic is encrypted. This includes the index files. So you still need to have the encryption key to look at snapshots and walk their trees.

> ZFS is not encrypted

It certainly can be (as of zfzol 0.8, and oracle zfs has its own encryption scheme).

https://github.com/zfsonlinux/zfs/pull/5769

If the backup server has access to the tree structure, then the backup isn't fully encrypted, or at least the metadata isn't. Now you need to have a conversation about what metadata is allowed to be stored unencrypted on the server and what metadata isn't. If you want a fully encrypted deduplicated backup system, I don't see how it's possible to prune without the ability to decrypt, because there's no way to know what needs to be pruned.

> (ZFS is not encrypted, but the principle is the same)

Neither.

It should be possible to have an index that says "this encrypted blob belongs to this particular snapshot". It introduces a "plain text" component of course, in that the snapshot information (name/date or whatever) is readable.

But then, I assume the way it is written, there are some blobs that belong to multiple snapshots, and some bits of blobs that belong to one snapshot but not another etc.

So given the implementation, it may not be possible, but it's still a drawback of this particular implementation that people should be aware of.

Couldn't the client just tell the server what to delete? Which would require only downloading an (encrypted) index of what data is in use, not downloading and reuploading the data itself.

I think that's what it does, but that downloading of encrypted indexes and comparing them takes time. Which is what was complained about.

> Secondly, I've set up an "append only" system, where my various hosts can append to their own backups, but not overwrite or delete them.

To my knowledge that's not possible with restic. You can either have append only (using for example restic/rest-server) or pruning. But not both.

My solution so far is to use a backend like rsync.net. The backup uses the normal backup + prune routine. And rsync on their server side creates daily undeleteable snapshots of all files and deduplicates the encrypted files in the process without the need to look into them. For me that's seems like a reasonable compromise for now.

I created my own restic backend using Nginx, which allows two different users to authenticate against a repo. The host being backed up knows the credentials that allow it to append. A different host knows the pruning credentials and performs that task.

I posted this in another comment, but I did this as follows (I'm the author) - https://www.grepular.com/Nginx_Restic_Backend

Another major problem is that it does not support compression: all files are stored uncompressed on the backup server (even though they are deduplicated) (sic).

You know, I actually assumed it had compression. It looks like it will get it eventually though, so I'm not too worried. That said, I can get some benefits by compressing some of my backups myself. For example, I currently backup my postgres dbs with cron jobs like:

$ pg_dump dbname | restic backup --stdin --stdin-filename dbname.dump --tag dbname -q

With restoring as simple as:

$ restic dump latest --tag dbname dbname.dump | psql dbname

I should probably stick a gzip/gunzip between those two commands.

If you stick a cheese up in there, restic will probably lose some of its ability to deduplicate.

If it does compression, then it's not an opaque process to it. But if you do it, you are essentially obfuscating the structure of the data.

Good point. I'll stay away from cheesing up my commands and wait for it to be built in.

What is "cheesing up"?

It's what Google speech-to-text thinks I meant when I said "gzip" into my phone.

Ha, I was thinking there's a new gzip competitor I needed to track down called cheese.

gzip with the —rsyncable flag will still allow for deduplication. It uses a content-aware rolling checksum and resets the compression dictionary whenever that checksum has a certain number of zeros.

The result is a valid gzip file with a few percent less compression, but friendly to deduplication (which is what rsync does).

Yes, that is the one thing that keeps me from using Restic as my backup solution.

Unfortunately it does not seem like it's a prioritised feature given that it has been planned since late 2014.

If this is a concern, why not delegate this to the file system level - just stick backups on a compressed volume and be done with it, no?

No.

You cannot stick a compression step before restic, because it will mess up the deduplication - the chunker will not find the same chunks, and will not be able to find out which files have same contents (not as easily at least).

And you cannot stick it after restic, compressing the backup archives (or storing them on a compressing filesystem), because those are encrypted, and any encrypted file has high entropy and does not compress. (It doesn't help that restic makes the encrypotion mandatory).

So no, compression really needs to be an integral step in a backup system. Restic is definitely lacking in this regard.

Ah, you are right. I forgot it was encrypting the backups.

There's an open PR to optimize prune-- essentially it lets you decide a percentage of waste per chunk you'll allow before repacking it. https://github.com/restic/restic/issues/2162

Meet this guy!: https://news.ycombinator.com/item?id=21411467

Reply on news.ycombinator.com