Sanoid – Replication and Backups with ZFS | Hacker News Reader

Sanoid – Replication and Backups with ZFS

github.com

125 pointsmkj54 comments

This is the piece I was looking for:

"Which would push-replicate the specified ZFS filesystem from the local host to remotehost over an SSH tunnel ..."

    syncoid root@remotehost:data/images/vm backup/images/vm

Now, if only there were a cloud storage provider that you could directly zfs-send to ...

If only ...

If only there were one that were a reasonable price increment on top of raw storage, like 1.5x or even 2x, instead of 5x what, say, S3 or B2 costs.

S3 currently[1] costs 2.3c per GB, per month and there are additional costs for various transfer and usage behaviors.

rsync.net pricing is 2.5c per GB, per month with no other charges possible, regardless of usage.

I feel like we're in the ballpark there ...

As a bonus, you can do things like this:

    ssh user@rsync.net rclone s3:/some/bucket rsync/home/dir

    ssh user@rsync.net rclone gdrive:/blah s3:/some/bucket

... which is detailed here[2].

[1] https://aws.amazon.com/s3/pricing/

[2] https://www.rsync.net/products/universal.html

Huh. I only use B2 (which is 0.5c per GB), and I had simply assumed that S3 was in roughly the same ballpark. Ok then!

Remote access to a ZFS gives you much different guarantees from your average S3-lookalike, so an integer price difference does not seem unreasonable to me.

2 is an integer! 3 is too...

Feel free to start a competitor…

It's probably easier and cheaper to just use Sanoid though.

[edit, of course moments after posting this I found this link https://www.rsync.net/products/zfsintro.html in the Arstechnica article. Still can't find it linked from the site though...]

> Now, if only there were a cloud storage provider that you could directly zfs-send to ...

Is there something on rsync.net's site that describes its support for zfs send?

I've found this old thread on HN referring to an Arstechnica article:

https://news.ycombinator.com/item?id=10751269

and a reference to it on IXSystem's site:

https://www.ixsystems.com/community/threads/rsync-net-now-su...

but they're both back in 2015.

I can't find anything on the site, https://www.rsync.net/index.html, but I may be reading right past it.

(As an aside, I'm grateful they've finally gotten over their absurd pricing notation)

I've been interested into rsync.net. How does it work if I want my remote ZFS dataset to be encrypted without leaving the key on the remote server?

"I've been interested into rsync.net. How does it work if I want my remote ZFS dataset to be encrypted without leaving the key on the remote server?"

If you have multiple terabytes of data, zfs-send is a very efficient (and elegant) way to transfer them to rsync.net.

We can give you your own zpool, with root, running the latest production/stable ZoL code - so you will have encryption and raw send, etc.

However ...

If you have hundreds of GB up to ... one or two TB ... we paste this suggestion to you instead:

I think you should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key to ... rsync.net has no access to the data.

The borg website is here:

https://borgbackup.readthedocs.io/en/stable/

and a good description of how it works and why you should use it is here:

https://www.stavros.io/posts/holy-grail-backups/

... and our own page with details of our attic/borg support, along with a deeply discounted signup link, is here:

http://rsync.net/products/borg.html

https://github.com/openzfs/zfs/commit/b52563034230b35f0562b6...

"The last addition is the ability to do raw, encrypted sends and receives. The idea here is to send raw encrypted and compressed data and receive it exactly as is on a backup system. This means that the dataset on the receiving system is protected using the same user key that is in use on the sending side. By doing so, datasets can be efficiently backed up to an untrusted system without fear of data being compromised."

Unfortunately it's taking a long time for the feature to propagate through the ZFS implementations. I don't know if rsync.net's servers support it.

Which remote server? You would only need the key at the remote server, if you wanted to mount the snapshot?

I have an Ubuntu server with a ZFS pool that I want to rebuild as a FreeNAS/TrueNAS server. Can I use something like rsync.net to push my datasets up as a backup, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets? Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about? (that come into play when pushing/pulling datasets)

> Can I use something like rsync.net to > push my datasets up as a backup

Yes, but depending on the size/value of the dataset, rsync might not be the best option.

Even after recent price reduction (Yay!), and with a hn discount (Yay!) - I still consider rsync.net as a personal backup service a little overkill/expensive past 100 GB or so. And likely you could just handle anything on the order of 1 to 2 tb locally anyway (spare disks).

You technically could use it for 10+tb of data... But uploading and downloading is going to be slow unless you have a real uplink (ie at least 1gbps).

And if you did, unless you need filesystem level Metadata - rsync might be a better option than zfs send/receive (even though zfs is cool).

>, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets?

Again, if you just need the data back - you might be well served with rsync or restic (both choices should be orthogonal to using rsync).

> Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about?

The world is divided in Oracle and not-Oracle - AFAIK there shouldn't be major differences between those two - that I believe now both follow upstream zfsonlinux.

However - I believe rsync.net still doesn't support zfsonlinux/openzfs encryption (zfsonlinux 0.8+). While locally it's possible to encrypt the disk "under" zfs... Native encryption is tempting.

And you would of course never send anyone data without encrypting it?

The switch of FreeBSD's ZFS master being ZoL is still work in progress, and is not the default ZFS in FreeBSD yet. (FreeNAS being FreeBSD + Configuration + Management Software).

A fair number of the ZoL 0.8 features (encryption, allocation classes being two prominent examples) are not available currently for FreeBSD

Allocation classes are available in FreeBSD CURRENT snapshots.

I just did this last weekend, except I have 2 local servers that can hold all my data in a pinch.

I tried to directly import the Linux created pools into freenas, but they are read only due to a feature property that’s only supported by ZoL.

In read only mode I sent everything to the backup Linux server, destroyed the main pool and sent the whole thing back to the Freenas setup.

Your pools may have incompatible features in use that would prevent this. I recently set up an Ubuntu (19.10 desktop) box and cannot recv it's datasets on FreeNAS.

For now I'm just doing rsync of the snapshot directories, not upgrading the Ubuntu pool, and waiting on FreeNAS feature support to match.

Just out of curiosity why do you want to rebuild this as a FreeNAS server?

I should also mention that this is actually an Ubuntu VM running in Hyper-V on a Windows Server 2012 host... don’t ask! It needs to be rebuilt one way or another (and it is running on an HP Microserver N54L!)

As for FreeNAS, I feel it gives the closest thing to a Synology NAS in terms of manageability through a GUI, etc, while also supporting ZFS.

Ahh, I don't have the hardware to run zfs in a VM, for home I run a big zfs array on bare metal and also host some vms next to it. If I were running zfs in a VM itself I would be more open to freenas.

Maintaining freenas and trying to run some other virtualized apps next to it was not good for me.

like rsync.net ? this is just storage.

The previous user runs rsync.net, he's pushing his service.

Totally missed that.

Sanoid seems like a great tool. It was at the top of my list when I was looking for this kind of solution.

I ended up going with zfs_autobackup - https://github.com/psy0rz/zfs_autobackup because it doesn't need anything set up on the computer to backup. Just a script on the backup server that does pull backups.

I'm not trying to hijack the thread but I've often appreciated links to similar software in threads like this.

The issue I have with that is that if everything is done by the backup server, it has the ability to both create and destroy (prune) datasets (snapshots), which means it has the ability to erase all of the data and all of the backups.

One thing I like about Sanoid/Syncoid is that if you set up Sanoid on both sides and use Syncoid to pull, the backup server only requires the ability to send snapshots (which you can configure with "zfs allow"). The backup server also doesn't require root in this case. This means neither server can delete the other's data.

And you still get all the automatic snapshot creation (done on the primary server) and pruning (done on both servers).

sanoid works great, used it for quite some time and the creator is actively involved in the zfs community

in the spirit of link sharing for other tools, I've been moving many of my personal infrastructure & tools to golang-based apps, and I started using this for zfs replication

https://github.com/zrepl/zrepl

No issues thus far in about 6mo of usage for local-and-remote snapshots

For FreeBSD users, it's available from ports:

https://www.freshports.org/sysutils/sanoid/

(I'm the port maintainer, open to improvements)

Only tangentially related, but seems like a good thread to ask:

Does ZFS (or any other FOSS file system) have a setup like Isilon or NetApp, where you could have any 2 disks in a box of 8 or any box out of say 6 die, and still not lose anything other than bandwidth?

I would say ZFS would be a lot closer to NetApp.

For Isilon, a better comparison would be Ceph. With Ceph you can have a cluster of 10s or 100s of servers, and you can lose 1 disk, or 1 server, or 1 rack or servers and it will recover very quickly.

As long as your minimum requirements for your type of replication or erasure coding profile are satisfied, your data stays safe and available.

RAIDZ2 / RAID6 (through linux md-raid, or anything else) both handle the 'lose 2 out of N+2 disks' case.

If you want something that deals with multiple tiers of fault tolerance, you should be looking at distributed filesystems (eg. Ceph) rather than single-system filesystems like ZFS and btrfs. Also keep an eye out for erasure coding as a keyword, since that usually implies more control over the degree of redundancy than a handful of RAID levels.

raidz2 for RAID6 like support where 2 disks can die.

This looks pretty solid. On one of my machines I'm using "zfs-auto-snapshot", which works great for something I can just install and not have to manage, for getting some history. For replication of snapshots, I haven't done that recently, but in the past I just used some scripts I had written to do one-time migrations. That was a decade ago. This looks much more capable than those scripts.

Yeah, there are base templates for configuration and they can be overridden on a single (or recursively on all children of a) dataset where needed.

Oh my. It's perl. Might be worth noting.

For those that actually use perl - I know perl has had (fairly sane?) package management for a while... Is it reasonable to use system perl version in anger?

I would typically reach for a version manager/virtual env with python or ruby - do you, with perl as well?

I'm fairly happy with asdf day to day.

https://asdf-vm.com/

Great piece of software; the templating model for configuration is neat and works out well for my case.

I love using Sanoid+Syncoid for keeping my laptop (with Xubuntu on ZFS) backed up to my NAS (running FreeNAS). In addition to the periodic snapshots Sanoid manages, I also have a small script (shameless plug [0]) that hooks into APT to take snapshots right before upgrades occur. It's already saved me decent chunk of time that would've been spent rebuilding from a botched upgrade.

0: https://github.com/paxswill/zfs-apt-snapshot

I wanted to use this tool for backing up my laptop to my NAS, however it only allows SSH to be used for data transfer, which is a real bottleneck for speed. It basically means you can only transfer as fast as one core can decrypt or encrypt data on either end of the pipe.

I ended up using https://github.com/oetiker/znapzend instead, because it can use mbuffer for the actual data transfer, which is much faster.

Is SSH speed a real limitation you’ve measured? I was able to easily saturate a 1 Gbps link for days using ZFS send over SSH with an Intel Atom when initializing my remote backups.

Yes. I have an Atom on one end of the pipe as well, but it's over 10Gbe and to a 6 drive mirrored and striped vdev. Using SSH as the transport it would only approach 100MB/s, using mbuffer it can max out the storage at about 250-275MB/s.

I've used a few different ZFS snapshot management tools, the one feature I'm missing is something to assist with data retrieval. I'd like to be able to specify a dataset, path and date/time and the tool would mount the closest snapshot to the specified time and pull the appropriate data for me.

Not exactly what you're asking I think, but in Solaris-derived OSs the snapshots are exposed by default as previous versions in windows.

You can also use SAMBA to expose the snapshots as previous versions in windows.

SAMBA general documentation: https://www.samba.org/samba/docs/current/man-html/vfs_shadow...

Randomly found ZFS specific: https://blog.chaospixel.com/linux/2017/09/zfs-auto-snapshots...

Another one, which creates backups on locally connected backup pool — https://github.com/reconquest/zeus

Is this a good fit for an on-prem small data center backup solution?

I've been using it for my daily zfs snapshotting needs for a year. Rock solid software. Thanks.

sanoid is great. I asked how to stop sanoid from snapshotting docker created datasets in rpool/var/lib/docker when rpool has the recursive flag, and they added a feature to do it the next day.

What advantages does it have over zrep?

yawn. hasn’t this been done a million times over?

Please don't be a jerk on HN. Also, note this guideline:

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

https://news.ycombinator.com/newsguidelines.html

in my defense, i wasn’t dismissing his work. i was wondering why it merits HN attention. it’s completely ordinary stuff. the world needs ordinary stuff but this isn’t buzznews “the 3 surprising tricks about zfs snapshots”.

out of all the various zfs snapshot tools, this is one of the better ones, and the creator is also an active community member on various tech forums

Reply on news.ycombinator.com