Hacker News Reader

Comment by rsync | Hacker News Reader

I gave a talk on this subject at CU last year and have, one way or another, spent my entire professional life thinking about this topic.

I agree wholeheartedly that simple systems have less downtime.

I would like to add a line from the talk that I give:

Simple systems fail in boring ways. Complex systems fail in fascinating, unexpected ways.

rsync.net storage arrays typically have multi-hundred day uptimes. But across our entire network, our actual aggregate uptime is unimpressive - perhaps 99.99% ? Our SLA dictates 99.95.

But when they do fail they fail in very, very boring ways. There are usually zero decisions to make in response to a failure.

View on HN

"simple systems have less downtime"

That is not much of an insight. It is as insightful as saying "water is kinda wet.". Well... sure it is.

What we need to deal with, is not "make a simplest system".

Rather, we need to deal with: "build a system that does A, B, C, ... and so on". Now, if you can do all of the above and make it simple... awesome. But if you cannot do all of the above, but the system is simple.... that is useless.

"Rather, we need to deal with: "build a system that does A, B, C, ... and so on". Now, if you can do all of the above and make it simple... awesome."

I agree with this. It is a good point.

Some systems and protocols have very difficult requirements to meet and different constituencies driving those requirements - it's not always possible to implement simple and elegant solutions.

beware: for most is not that obvious. For most developers modern, extensible, layered comes before simple.

rsync.net is built on ZFS which is anything but simple. Having worked with multi-petabyte ZFS systems, you can run into some seriously hard to track down issues.

"rsync.net is built on ZFS which is anything but simple. Having worked with multi-petabyte ZFS systems, you can run into some seriously hard to track down issues."

Obviously I have a few things to say about this ...

First, zfs was considered production worthy on FreeBSD and put into fairly widespread use as early as ... 2008 ? 2009 ? We could have made very good use of it as we were running into all kinds of limits and corner cases with UFS2 and very large filesystems with hundreds of millions of inodes.

But we waited until late 2012 to do our first deployment and it took us about six years to finally deprecate the last UFS2 systems. We did this out of an abundance of caution and a desire to see things shake themselves out over several major releases of FreeBSD.

As for the complexity of ZFS, our previous architecture had 3ware RAID cards and all of their firmware and complexity sitting between the drives and the OS. In a way there was a beautiful elegance (in my opinion) in giving the OS a single drive:

    newfs /dev/da0

.... and that single drive just happened to be 40 TB in size and the OS has no idea what's going on underneath ... but there is a lot of complexity inside a full-blown RAID card and a lot of ways for drives to interact weirdly with it - especially when the drive is X years newer than the latest firmware for the card.

I find that on the very deepest, hardware level, handing over raw disks to ZFS and letting it manage them is simpler. We remove a fairly complex piece of hardware and accompanying firmware (since our HBAs are "degraded" to dumb "IT" mode).

Further, all of the "bolt-ons" of UFS2 that we absolutely relied on, such as quotas and snapshots, are elegantly built into the filesystem from the very lowest levels.

We ran rsync.net (and its predecessor) on UFS2 for 11-12 years and I can tell you from the perspective of both day to day management and middle of night firefighting, ZFS has made our lives much simpler.

Is ZFS more or less complicated than mdraid+LVM+ext4?

I'd say it's more complicated even for simple ZFS pools without any redundancy or fancy features. My favorite illustration on this topic would be figure 2 from "Reliability Analysis of ZFS" (PDF): http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf

I still miss some of the features that are hard (if not impossible) to replicate in layered storage systems, like Merkle tree checksumming, but near-nightmare that low-level debugging of ZFS was actually turned me away from that filesystem.

sorry for offtopic.

there a broken link that I was interested to read on. its on https://www.rsync.net/resources/howto/rsync.html ctrl+f "rsync snapshots are detailed here"

I think given the topic, someone is expected to figure it out on their own (remember linux cake?), but I feel a bit lazy after a work day.. sorry

Sorry - that is in purpose ...

The definitive page for rsync snapshots has been, and always will be, here:

http://www.mikerubel.org/computers/rsync_snapshots/

... we actually don't want people to do rsync snapshots anymore because ZFS snapshots are much more efficient and use up less of their rsync.net account.

If you change one bit of a file, the rsync snapshot method will cost the entire size of that file, since it's a hard link and is either broken or not broken.

But if you just do a "dumb" sync to us and change one bit in the file, your ZFS snapshot will take up just one more bit.

So not only do you get a much simpler backup script - you can really just do a dumb sync to us and let us rotate your snapshots - but you get more efficient space usage for changed files.

Thanks for the link and explanations, appreciate it!

I've read that page (i linked) inattentively, re-read 3 times that bold text next to the broken link and still asked that question... :facepalm:

Reply on news.ycombinator.com