Ask HN: ZFS on FreeBSD vs. expensive NAS for 100 TB backup over NFS? | Hacker News Reader

Ask HN: ZFS on FreeBSD vs. expensive NAS for 100 TB backup over NFS?

19 pointsreacharavindh22 comments

Fellow HNers! I have a chance to upgrade our backup server that is an aging NetApp Filer costing us big money. Is it a good idea to setup FreeBSD-ZFS on Directly attached JBODs and use it over NFS?

Or are there any practical reasons I should look into enterprise NAS vendors(NetApp, EmC etc al)?

I imagine the server and disks I'd buy would come with support, and I can replace them as needed. It's usecase is stupidly simple: 4 NFS mount points to a single host that rsyncs our primary storage every night. One backup admin user and nothing fancy.

Any experiences, suggestions before I step into building this thing?

View on HN

Programmer's opinion here: I'm not a sysadmin, just like to do light HomeLab style hobby work at home.

I can't provide a counterpoint for why you might consider a proprietary NAS, but I can tell you that I'm 90% sold on FreeNAS. I've built two servers running them from commodity hardware and never once had a problem with it.

I use rsync to backup to my older NAS no functioning specifically for that purpose. It's a ReadyNAS x86. I moved away from that because I wanted more functionality and redundancy.

I've had 2 drive failures on 5-disk setup. Live rebuild in degraded status worked as well as it should have. I also use the SSD caching feature which sped things up a bit for frequently used files.

My only qualm about the product is that whole versioning fiasco. I actually almost got bit by it while building a server for my parents, but the install wouldn't take so we used the older version. But by the grace of luck we didn't use the bad version.

I use separate hardware for my service host which talks over NAS. I just use the FreeNAS boxen for storage. Learned that lesson!

Thanks for the comment. I'm a sysadmin albeit only recently changed into this role. FreeNAS was my first choice when I thought about building this solution, however, instinctively, it feels over engineered for my needs. Sure, if I want a NAS with GUI, Jails and running VMs etc, I'll run to FreeNAS. But, I'd like a minimal core (FreeBSD + ZFS + NFS + rsync) which lowers the failure surface. I will write a few scripts to keep an eye on the health of the server, and that'll be it.

Good to know that your NAS survived a 2/5 disk failure.

2 things:

- IMO, having owned a couple readynas(es) in the past, FreeNAS is a good compromise between ease of setup and functionality. To me, it does not feel over-engineered. In fact, it feels like it gives you enough to do everyday stuff easily through the UI, and drop into ssh if necessary. I've only had to do this a few times. To wit: I would take a stock FreeNAS over manual setup anyday.

- The 2 disk failures I've experienced were independent of each other. I have not experienced a simultaneous failure of 2 drives. I have preventative measures in place, namely in the form of a hot-spare for the pool and an aggressive resilver schedule that runs weekly.

Why would you use NFS when you can use rsync and for files something like MinioFS?

When NFS goes down, its a huge PITA. I’d rather have something that deals with disconnects better.

While I would personally prefer FreeNAS (or just to roll my own), remember that you're supposed to be planning for the next person in your position. Sometimes a commercial appliance really is the best way to go. If you do roll your own or set up FreeNAS you need to build lots of docs and make sure they're stored somewhere the next guy will find them.

Ultimately it depends on the company, it's just a compromise I have had to make before so I thought it worth mentioning.

Good point. I'm thinking of this custom solution only because of the simplicity of this requirement(NFS backup). I'm planning to prepare a thorough documentation of what I did to build this solution and how to maintain it.

There's a storage startup that posts their hardware designs openly on here. I forget the name though.

Edit: https://www.backblaze.com/blog/open-source-data-storage-serv... Cost is $0.036/GB

Alas it's ext4. You could do XFS if you want something in the mainline Linux kernel with a long history, btrfs or ZFS-on-Linux if you're happy with one or the other, or just run ZFS/BSD as you mention if you've got BSD skills.

Edit: I know it's not the done thing on HN to complain about downvotes, but why on earth would anyone downvote an entirely informational post?

With respect, Backblaze can’t be used as a NFS server replacement.

It’s fine if you want to use them for storing backups with their proprietary protocol, or use S3-style bucket storage on their B3 service, but that is totally not the same thing.

It's principally a hardware design. You can use it with whatever software you like.

Take a close look at that hardware design — it is very much intended to be used exclusively for the purpose it was designed for, with their proprietary HTTP application on top. They are tightly coupled — it is very hard to separate the hardware design from the software that would be running on it.

If you were to design a more generally useful storage server, it would look a lot more like the FreeNAS or TrueNAS boxes than it would the Backblaze boxes.

The biggest reason going for an enterprise vendor is support! As long you pay them big money you will get something that just works without worries. If something should go wrong, they will be on your door ASAP and fix the problem for you. Very convenient.

If you have expertise and is willing to do self support, creating your own DAS may be fine. Just keep in mind though, that things can go really wrong at the worst time when you are busy with something important.

A lot of businesses do bad resource management and don't do proper planning. This can hurt you.

Because we think most major storage vendors are way too expensive for us, and we don't have resources for self support, we have landed on using Aws S3 and Aws Glacier.

At home I've been running a FreeBSD ZFS system for 8 years. Oh boy I have had many issues over the years because of bad hardware, cheap SATA cables, consumer grade SATA controllers, consumer grade network cards, consumer grade hot swap bays, consumer grade motherboards. But now I use only enterprise stuff, except from the hard drives, which has been very stable for the last two years.

True. This is the reason why our primary storage is an Isilon cluster supported by EMC. This is something we can't afford to have downtime with.

The NAS I was asking about is a passive backup server. If something goes wrong and ZFS asks me nicely to do do a clean boot or a scrub, we'll be perfectly okay. Even if it is down for a few days, we can catch up from snapshots on the primary storage.

Good point about going for enterprise grade hardware instead of putting together cheap parts. Duly noted. I will buy a decent server with good support and lots of ECC memory.

I recommend taking a look on borg backup (https://borgbackup.readthedocs.io/) or restic (https://github.com/restic/restic) instead of rsync... and use snapshots for zfs.

I use BorgBackup for backing up my personal website (in the order of a few MBs) to learn. But, my use case in this quest is to build a long-term backup solution that we can rely on. (10 years+). My concern is Borgbackup being updated so frequently and being actively developed to add more features. What is the chance that I can read a backup that I make today 10 years from now? I do NOT want to maintain a directory full of borgbackup binaries of different versions just to be able to access older backups. Same goes with running unupdated version of Borgbackup for 10 years.. Hence my interest to keep things stupidly simple with NFS --> ZFS server.

Well, that is understandable... but i think, that rsync has major drawbacks: No history - overwriting means destroying (e.g. ransomware) || No deduplication - restic transfers blockwise and only once || No encryption - access to backup means access to all data || No error protection - how to verify backup success?

In my opinion a good backup should contain a verified restore strategy :-)

You should also consider sending and receiving snapshots, when using zfs (https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm...). Zfs also supports deduplication, but FreeNAS devs do not recommend using it unless you have a massive amount of RAM and you know what you are doing.

Well, if not FreeBSD, you could take a look at OmniOS (https://omnios.omniti.com/). There also is a project with a web based administration for this os called napp-it, that could be used as is or, in your case as "minimal installation reference guide" (http://www.napp-it.org/index_en.html)

Looks very interesting. However, I'm not familiar with Illumos/OmniOS to begin with. Even though ZFS is still ZFS, I'd be concerned about dealing with operation of an OS that I'm not familiar with. I've played with FreeBSD for a few years, and I know its community help is great along with the wealth of StackOverflow responses. I'd try OmniOS for a smaller scale private project to learn it.

This is a perfect use case for ZFS. We run a number of ZoL (ZFS on Linux) filers in various backup/archival roles, and the cost savings and flexibility are great. These range in size from about 100TB to 1.6PB.

Since you are a single dev with limited experience with this, hopefully I can think of a few common gotchyas.

1) Many disagree here, but I do not think ZFS is ready to run as primary storage for say backing a vmware cluster. We've tried in the past, and there are far too many "reboot the server" style bugs when you get into heavy load. Many have been fixed, but simple things like certain types of drive failures ZFS on FreeBSD still can have issues. We've even experienced this on IllumOS/Nexenta as well. Data integrity will let you sleep at night, but don't expect 100% availability comparable to a Netapp or the like. This is why we use ZFS in a backups/archival role only, where 100% availability is not required.

2) Understand your I/O needs and how that will impact your choice of mirrored vdevs over raidz(2) vdevs. You will likely not get the space efficiency you are mentally calculating either, which is something to keep in mind. I'd take a look at the raidz efficiency calculations[0] and keep in mind you should never fill a zpool beyond 90% capacity or so.

3) Shouldn't need to be said - but run ECC memory. RAM is cheap, buy a lot.

4) For rsync based backups (assuming backing up actual user directories and such) you will have potential for a lot of small file writes. I do suggest a small SLOG (write cache) ssd for these cases, as random writes against a raidz vdev consisting of spinning rust can be rather slow. I don't see a reason for any l2arc (read cache) here.

5) You're doing a one-off. Do not go exotic on the hardware. I recommend a standard 1U server with the LSI HBA of your choice, connected to an appropriately sized SAS JBOD enclosure. We've had good luck with both HGST 4U/60 drive units, and Supermicro 4U/45 drive units depending on the density we need and if top-load is a good choice for a particular facility. I suggest avoiding top-load as your first deployment, assuming you are not space constrained. There are plenty of VARs out there who can give you a pre-packaged deal.

6) Due to the way ZFS stripes over vdevs, it's better to build out capacity up-front vs. add later if performance is a concern. This is likely much less of an issue for your use-case but is important for planning. For example if you know you will need an additional 100TB in 12mo, buy it all at once and add it to the pool day one. Don't trickle in new vdevs 8 spindles at a time.

7) This is probably the largest pain point for ZFS right now - you will have to build tooling to properly monitor and administer it. This isn't hard, but is something to very easily forget or have break and not notice if you're a single man "ops" team as your second responsibility. Remember to monitor zpool space usage and to alert early to add capacity. Stuff like a hot spares while getting better I wouldn't trust to automatically Just Work(tm) yet, so ensure you have robust drive/pool health monitoring.

8) Snapshots! Remember to use them, they are magic. znapzend is a decent tool for getting schedules going.

9) Sounds like you won't need it now, but zfs send/receive combined with snapshots are one of those life changing technologies like Tivo I don't think I could live without these days.

Overall ZFS is great, if you enjoy a little hacking this should be a fun project for you.

Edit: links, woops.

[0] https://docs.google.com/spreadsheets/d/1pdu_X2tR4ztF6_HLtJ-D...

I'm just going to put this out there since we're discussing ZFS. There's an open ZFS conference in Norwalk CT in April: https://www.eventbrite.com/e/zfs-user-conference-2018-ticket...

ixsystems.com

Makers of FreeNAS/TrueNAS. Good team. Great support.

I was thinking of using FreeBSD for a large filer as well, but then I learned that you can run ZFS on Ubuntu, which should have better driver support than FreeBSD (at least I would think so). So you may want to consider Ubuntu with ZFS instead. https://www.howtogeek.com/272220/how-to-install-and-use-zfs-...

"Better" is a little subjective. Linux has more drivers, true. But you only need one driver, for the HBA you use, and is the quality of that one driver better? I bought an LSI HBA specifically because the FreeBSD driver for it is known to be good (and it also works well with Linux).

Drivers aside, ZFS on Linux is rather more rough around the edges than on FreeBSD. Having used both, I'd place more trust in ZFS on FreeBSD. While I've not had dataloss on either platform, I have seen odd glitches on Linux which required rebooting to resolve, like zpool getting stuck in D state after some pool operation, and it failing to mount filesystems after rebooting until I manually reset the mountpoints for little reason I could see, and excessive memory usage which has frozen the machine on a few occasions when under heavy load. It's also rather more featureful on FreeBSD; the Linux implementation has a number of annoying restrictions and missing functionality which aren't essential for the basics, but make it much more pleasant to use. Like not requiring root, priv delegation, NFSv4 ACLs, NFS export, ARC integration into the kernel memory management.

Reply on news.ycombinator.com