Show HN: NFreezer – Encrypted-at-rest remote backup tool
github.com
github.com
I created nFreezer initially for my needs, because when doing remote backups (especially on servers on which we never have physical access), it's hard to 100% trust the destination server.
Being curious: how do you usually do remote backups of your important files?
--> With usual solutions, even if you use SSH/SFTP and an encrypted partition on destination, there will be a short time during which the data will be unencrypted on the remote server, just before being written to disk and before arriving to the encrypted filesystem layer.
Thus this software nFreezer: the data is never decrypted on the remote server.
How do you work with this?
It can run over SSH with the borg binary on the remote server or it can run in an SFTP mode with nothing installed on the destination.
If you want some assurance, CERN are deploying restic:
When using with SFTP, do Restic binaries need to be installed on both local and remote, or just local?
But still there is one thing (at least) that can be of interest to some people with nFreezer: it is very simple, it does the job in only 249 lines of code. You can then read the FULL source code in a few hours, to see if you trust it or not. See here: https://github.com/josephernest/nfreezer/blob/master/nfreeze...
If I want to do this with the source code of the tool you mentioned, I would have to spend at least one full week. (this is normal: this program has 100 times more features).
The key point is: if you're looking for a solution for which you don't want to trust a remote server, then you probably don't want to trust the backup tool of a random internet person either. And you probably want to read the source code of the program.
So having only < 300 lines of code to read in a single .py can be an advantage.
On the other hand, missing features in the name of small code size is not a great advantage, if those missing features make my backup less reliable or quick or compact. And maybe I end up backing up less stuff because (say) the simple tool does not handle deduplication or compression as effectively as the complicated one, and so is not viable for me. This would be a net loss.
Be careful, I am just saying my perspective. You have, of course, the entire right to decide what are your priorities and which tool is better for you.
I really appreciate this and find this very interesting.
I would be happy to give you a free dev/test account at rsync.net if that would help you continue this development.
[...]
with open(f2, 'wb') as f, src_cm.open(chunkid.hex(), 'rb') as g:
decrypt(g, pwd=encryptionpwd, out=f)
[...]
def decrypt(f=None, s=None, pwd=None, out=None):
[...]
while True:
block = f.read(BLOCKSIZE)
if not block:
break
out.write(cipher.decrypt(block))
try:
cipher.verify(tag)
except ValueError:
print('Incorrect key or file corrupted.')
As some HN crypto celebrity said some time ago, if you write "AES" in your code then you're wrong. You MUST use misuse-resistant libraries unless you know exactly what you're doing.
TL;DR: your crypto is broken, use NaCl instead of doing it yourself.
except ValueError:
print('Incorrect key or file corrupted.')
print('Incorrect key or file corrupted, will be deleted:', fn)
os.remove(fn)
exit(...)
That's the thing: the script in its current version is incorrect, and even doing that won't be a perfect solution. That's why other people are saying that other softwares, with large usage and that can do more than what nFreezer can do, should be analyzed before trying to do it your own way.
It's good to not rely on anyone else, but crypto is the one domain where you can't have "good enough" -- it's either correct, or it's not.
> Another option would be to do a first pass of decrypt(), without writing anything to disk, then get the tag, verify it, and then if ok, redo the whole decryption with writing on disk this time
Yep, that's the way: do the decrypting in memory, or in /tmp, verify the tag, and only after you can put the file where it belongs. I just checked the API of the crypto module, and there's a `decrypt_and_verify` that should do it properly.
Of course that's problematic especially for big files, so what you want to do is chunk the files, encrypt the chunks separately and store the file as a list of such chunks.
The step after is to use Content-Defined Chunking, ie chunking based on the content of the file. This way when a big file modifies only the chunk around the modification will change, the rest of the file will be chucked exactly the same way. So you don't need to store the full content of each version of the file, just a small-ish diff.
That's not a novel system, bup (https://github.com/bup/bup) kinda pioneered it... and as others have advised, restic, borg-backup and tarsnap do exactly that.
cipher = AES.new(key, AES.MODE_GCM, nonce)
while True:
block = f.read(16*1024*1024)
if not block:
break
out.write(cipher.encrypt(block))
cipher = AES.new(key, AES.MODE_GCM, nonce)
out.write(cipher.encrypt(f.read()))
You might find this unforunate in the naming, and .init(), .update(), etc. might have been better names to emphasize this.
So this shows that, in its current state, the chunking is just a "RAM-efficient" way to encrypt, but it writes exactly the same encrypted content, as if you did encrypt(...) in one pass. So as long as the file is under ~2^39 bits, it is fine (see https://csrc.nist.gov/publications/detail/sp/800-38d/final).
___
Then, there is another layer of chunking that would be possible, and that would add many benefits: even better deduplication, avoid to reencrypt a whole 10GB file if only a few bytes have changed. This would be an interesting addition, it's on the Todo list, and I know some other programs do it, of course.
But to clarify: this "content-chunking" is independent to the "RAM-efficient" one I use here.
If you want to continue the discussion (more convenient than here), you're very welcome to post a Github issue.
Thanks for your remarks, I appreciate it.
According to wikipedia, bup was released in 2010, 3 years after Tarsnap started doing this. (And Tarsnap wasn't the first either.)
You will probably be surprised, but I wasn't aware of this one when starting my project! In a way, this is fortunate, because it was an interesting journey to code this :)
I use rclone+restic because of google drive(=cheap unlimited storage) integration in rclone, and restic provides snapshotting on top.
The "key thing" that lead me to code my own is that with nearly all the solutions I tried, data was resent over network when a big file is moved to another dir+renamed, WHEN used in encrypted-at-rest mode.
It's the case for rsync (the --fuzzy only helps when renamed but stays in same dir), duplicity, and even rclone when in encrypted mode (see https://forum.rclone.org/t/can-not-use-track-renames-when-sy...: "Can not use –track-renames when syncing to and from Crypt remotes").
So I wanted to solve this problem, and it works. See point #3 of https://github.com/josephernest/nfreezer#features.
restic (and i suppose borg which is similar) solves this problem and goes even further by chunking your files, hashing and deduping the chunks - chunks with the same hash aren't resent. Great e.g. for backing up VM images or encrypted containers where only a small part of the file change - only that small part will be resent between snapshots. Chunking algo is "content-defined", can probabilistically quite efficiently detect shifted chunks and duplicated chunks across different files.
(Naturally, all this machinery will also handle the simple cases of renamed and duplicated files on your filesystem)
Indeed, Rclone does not handle renames when in encrypted mode, see https://forum.rclone.org/t/can-not-use-track-renames-when-sy...
Idem for duplicity (tests done).
TL;DR: see 3rd point of https://github.com/josephernest/nfreezer#features
___
Another interesting feature is that you can read/audit the full source code of nFreezer quickly: it's only 249 source lines of code, as of today.
This is the use case I mentioned: I often move files from one dir to another when working on audio- or video-production projects, and it can be dozains of gigabytes.
[0] https://forum.syncthing.net/t/rename-file/11091 [1] https://wiki.archlinux.org/index.php/Synchronization_and_bac...
My setup depends only on crappy home made bash scripts and standard utilities, and never leaks encryption keys or filesystem metadata off the local machine. I rsync my home directory with a luks encrypted filesystem image, split the image file into fixed sized chunks using the standard GNU split utility, and then rsync the encrypted split up chunks with a directory on a remote server. The remote server periodically syncs the encrypted image chunks with S3 using the usual aws cli tools, and runs a script to "cat" the chunks back into a remote copy of the whole image and display the overall md5 sum. Another machine off site keeps amazon honest by syncing the s3 bucket with a local copy, reassembling the encrypted filesystem image, and computing the md5 sum. An extra benefit is that I can use the filesystem image on the remote server over sshfs as the backing device for a locally loopback mounted luks partition to access or modify individual files without downloading the whole image or corrupting it, provided no more than one session is open at a time. Back when I was more paranoid, I used only a raw dmcrypt image rather than a luks encrypted image in this setup so that there would be no luks header to give it away and I could plausibly claim that it was one-time-pad encrypted data, for which I could construct a key to yield any plaintext of my choice. (Yes, I know I won't be such a smartass when it comes to a rubber hose attack.)
If you were using an untrusted server as a zfs send target, as of a few months ago, you can actually do an encrypted send to a remote without unlocking on the remote, which is pretty cool.
I keep my important files in a ~50GB veracrypt container then backup that to multiple locations including a local usb disk every day or so with a shell script.
dropbox nuked its copy once but it worked fine again after a name change (version number)
This is quite easy to accomplish with public key cryptography - just use a new, random AES key for each backup job, then encrypt that with a public key you keep around. The private key can be stored offline in cold storage.
> cloud backends are WIP
and I probably don't have good enough rust skills to help out myself practically.
In fact, if you encrypt the same file twice with rsa you get different cipher text.
No one encrypts a file directly with RSA. It’s the data encryption key that is encrypted .
Then, I create another mount on the physical disk and simply sync _that_ mount to multiple remote sources. Of course, this solution is fairly simple, and does not provide the features like file moves. Further, this approach potentially "leaks" some information, such as how many files I have, the approximate sizes of each file, etc.
This was set up a while ago; will definitely take a look at NFreezer to see if it's time for a refresh of my setup.
For example, restic with SFTP backend. I thought restic encrypts data client side before SSH transport. Am missing something?
Other tools: duplicacy, rclone, Borg (this might run on server), ... Duplicity does asymmetric and symmetric crypto, but no dedup. So if you change a file path, it “might “ be reencryptped.
about duplicity and rclone, they don't track file renames/moves in encrypted mode (https://forum.rclone.org/t/can-not-use-track-renames-when-sy...).
i haven't tested the two others (duplicacy, borg), but by reading other comments, it seems the second one has this feature.
To the point where in my organisation we're told to do application encryption on resources that really don't need it
Please read what encrypting at rest is for here: https://www.hln.com/encrypting-data-at-rest-on-servers-what-...
A security problem is still a security problem, encryption at rest is about physical security and encryption in transit is about moving data securely between locations
Your applications still need to be secure otherwise encryption won't save you