How I Store My 1's and 0′s: ZFS + Bargain HP Microserver = Joy
mocko.org.uk
mocko.org.uk
- For best performance with ZFS, you want a lot of RAM, and this unit will take 8GB of ECC RAM. You want ECC for data integrity in memory, as ZFS does nothing to prevent in-memory data corruption (there's an article on this here [pdf]: http://research.cs.wisc.edu/wind/Publications/zfs-corruption... )
- You probably want a few mirrors, not RAID-Z, if performance is an issue.
- You're better off with FreeBSD or Illumos kerneled distros (which have run ZFS for years, and have it in their mainline kernels), rather than Linux (which never will have ZFS in mainline for licensing reasons), for stability alone.
- You can get an IPMI card for this unit if you want remote manageability.
- There's an internal USB port if you want to boot off of that. It's kind of handy.
btrfs is still considered experimental, even though it's been in the mainline Linux kernel since 2009.
You might look into the KVM support in SmartOS if you want to run other OS's on top of Solaris/ZFS.
I'm interested in SmartOS but I've heard some things about it not being production ready. What's your view?
Also, being able to boot from an internal volume (not USB) is also crucial to me, as the alternative is simply not an option. (The "best" case, using a USB drive, is not an option as our server is in a shared rack and you run a risk of someone unplugging it).
You also need to ensure you have adequate ram for various cacahing, and CPU power as well for checksum calculation, among other things. It's not a lightweight FS. (in-memory corruption is an issue with all filesysems - just somewhat more-so with ZFS because it was designed specifically to assume you had reliable ram. You want ram to store the hash cache or whatever too....
One could chuck an SSD in there for cache, if memory is a limit, that should speed things up drastically.
And as with all raid-like systems - you want an appropriate number of hot-spares, cold spares, and a system that monitors it and acts appropriately, especially if you're going with huge drives on slow busses.
You want regularly scheduled scrubs, not too many snapshots, probably disable atime (noatime) to speed up those scrubs, and compression probably off....
dedup I'm still on the fence about - I leave it off, I can only see specific situations where it would be truly useful (dedup+verify)
Cheap consumer class SSD's are generally fine for L2ARC.
I'm using FreeNAS and 5x2TB raidz1 in the N36L and if I need more space, it's going to require some very careful planning.
That said, the hardware is a STEAL for the price, and FreeNAS + transmissiond + remote-gui rock my socks off. I'm strongly considering just buying a brand new one when I'm out of space on the N36L.
I wonder if the performance might be better if a good 16GB USB stick was used for the OS drive instead of an old laptop drive? The OS needs a lot of random access, but doesn't take up much space.
I also wonder why the author went with Ubuntu 10.04 LTS instead of 12.04 LTS, which would give him two more years of peace of mind. It's been a few weeks since 12.04 came out, so it's pretty stable. It does get kernel updates more often that I'd prefer, though, and GNOME 2 is gone.
My dev machines are running 12.04, though, and haven't run into any major issues yet (knock on wood); my daily driver netbook runs 12.10 because I like it like that.
I mean, let's just not leave it at idle claims:
btrfs changes in 3.4: http://kernelnewbies.org/LinuxChanges#head-556161b206bf626d6... 3.3: http://kernelnewbies.org/Linux_3.3#head-1f03b4babafb1049bea3... 3.2: http://kernelnewbies.org/Linux_3.2#head-f0a922e9c0ce6f48810d... 3.0: http://kernelnewbies.org/Linux_3.0#head-3e596e03408e1d32a7cc...
RAID5/6 barely missed 3.5, but should be in the pull request for 3.6 (other changes for 3.5: https://lkml.org/lkml/2012/6/1/160).
What's going on in XFS: http://lwn.net/Articles/476263/
Coverage from the 2012 filesystem/storage summit, day 1: http://lwn.net/Articles/490114/ Day 2: http://lwn.net/Articles/490501/
2011 summit, day 1: http://lwn.net/Articles/436871/ Day 2: http://lwn.net/Articles/437066/
Ext4 is still pretty active for a "done" filesystem, too: The last 12 months saw work on online resizing, support for bigger block sizes, a cleanup of mount options, ...
On the other hand, the only in-kernel Linux filesystem that can match ZFS's feature set (for example, resizing a RAID array while the filesystem is online) seems to be btrfs, which probably won't be marked stable for at least another year or two. The latest developments to XFS and ext4, though interesting, aren't particularly relevant if you're looking to build a server like what the article describes.
Still, thanks for the many interesting links!
Who cares? It is a file server not a desktop machine.
I started using OpenSolaris fileservers at home in 2007. I went through multiple upgrades since ten: 5x500GB, 7x750GB, and currently 7x1.5TB (all raidz).
I am just about to upgrade to my 4th server with 6x3TB in raidz2, which will be running FreeBSD this time.
Due to the sheer number of drives and continuous 24/7 operation, I experienced 4 drive failures over the years. Everytime ZFS has handled the failure gracefully, and I was able to swap the drive and rebuild the array without a hitch.
I take daily rotating snapshots - incredibly useful when you accidentally stuff.
I also run weekly scrubs, which have allowed me to witness 2 or 3 silent data corruptions which were automatically self-healed by ZFS (CKSUM column in zpool status).
I mostly use the file server to share videos & photos via NFS, and to store encrypted backups of my laptop. It has become so useful and practical that I started to use it as an iSCSI server as well to boot some diskless Windows test machines for GPU projects.
All in all, ZFS deserves all the praise you hear.
It supports almost everything out of the box, and there's very little configuration. It has 6 hot-swappable bays, and it allows for automatic expansion using their proprietary system, X-RAID 2. I currently have 4 500 GB drives, and 2 1 TB drives, and if I want to expand it, I just buy another 1 TB drive and swap out a 500 GB drive.
It also supports streaming protocols, including ReadyDLNA so I can play movies directly off my PS3. It also seamless supports TimeMachine for my Mac laptops. I really do love this thing.
It also runs Linux under the hood, and I've been able to configure CrashPlan for off-site backups of some key files too.
i got a pro recently to share/backup a dropbox share over a network. The sharing, raid and backup stuff is all standard with the linux based os which makes it nice and easy. the pro readynas boxes are also intel atom cpu so its simple to stick the linux dropbox client on it.
This HP box looks much better value than the readynas though, as long as you want to install the OS/raid etc yourself.
That said, there are some cons too. In my limited testing, ReadyNAS is not the fastest device out there so if you are looking the absolute fastest, they are probably not it.
The older Sparc based NAS devices do not get as much firmware love as the newer ones. So probably no IPv6 and GPT. Also, the older devices support only ext3 not ext4.
I found out last year that my particular NAS was due for a recall on the power supply. Unfortunately, this recall had been issued several years earlier and I just had not noticed it. TL:DR, ReadyNAS replaced it gratis even though the recall notice had expired several years prior.
Other than what I listed above, I can wholeheartedly recommend the ReadyNAS. Remember, this is your data we are talking about and it is sufficiently important to me that I'm okay spending a bit more money to make sure it is well protected.
So illumos remains the repository of record for ZFS -- with a close relationship with those working on ZFS on FreeBSD. While the Linux port is certainly a Good Thing, it does not reflect shift in the epicenter of ZFS development...
All my prepare and forget servers runs openbsd just because they have less people adding features (and bugs) and the few people there have awesome attention to quality (not just security) to the point of trading documentation bugs as well, bugs.
Granted it will move slower. And you may not have zfs...
It's too bad, because I'd planned a SmartOS+Ubuntu build for mine.
[1] http://wiki.smartos.org/display/DOC/SmartOS+Technical+FAQs#S...
ZFS with RAID-Z does block-level checksumming and automatic healing as you access your data. Combine that with a weekly scrub (touches every block so any silent bit flips are healed) and I can do away with my fears that my precious bits are rotting away.
I use it as a Time Capsule for my MacBook. Hooked up to gigabit ethernet when docked, backups are a breeze.
For my home setup I actually run FreeNAS in an ESXi environment. I have 4 different physical disks that I carve up space on and allocate to the FreeNAS VM. This allows me to snapshot upgrades on the base OS and when I'm testing an upgrade I can disconnect the ZFS pool - validate the upgrade went fine, and then reconnect the pool for a ZFS upgrade (if needed). The nice thing about this approach is you can physically move around system very easily if all you need to do is ship out the disk store - and since I have one beefy box for virtualization at home I have my storage system contained within which makes power and space a bit more efficient.
My suggestion to those who are considering this is if you stick about $500-$1000 into a BYOD system you can generally get a high end quad core system with 32GB of RAM and 3-5TB of disk space (with an SSD boot). At that rate I would carve up a few TB for backup and SAN (FreeNAS) and the rest would be for on-box VMs. The FreeNAS VM doesn't need more than 1 proc and about 2-4GB of RAM if you're dealing with a lot of file transfer. You can easily get away with 2GB.
Long story short: ESXi + FreeNAS = 1-box solution for most at home geeks. My motivation was that I was starting to have "box sprawl" and power consumption was getting a bit out of hand. I also run PFSense on my box as well - but in that regard I also have a low power physical system that acts as the primary gateway device in my network. But there's a PFSense running as a VM as well for failover when I do upgrades. Far better than any SOHO gear you can buy for far too much $$$.
At ~£150 after rebate (was around £120 when I bought mine) the MicroServer is an absolute steal, and ZFS is a dream to administer. Truly a match made in heaven.
What I am looking at doing is getting/building Solaris compatible box for my backup needs - that is a daunting task. But if I could do that I can run one of the OSS variants of Solaris - Joyent SmartOS, Nexenta etc..
I recommend at least trying ZFS out in a VM, I guarantee you'll be impressed by the versatility.
But know that to really use it, you want decent CPU speed, as much ECC ram as you can put in there, an SSD for cache (just one - doesn't have to match the raid size I don't think - it'l help a lot) and you still need to take into account all the normal raid cautious everyone ignores like rebuild times for 2TB sata2 drives -vs- failure rates, etc.... and if you want to use dedup without verify, that's your gamble.(smarter people than me say it's safe, but it just smells wrong)
With Debian, you can use the PPA as-is. This requires adding that to your /etc/apt/sources.list and manually adding the signing key with apt-key.
Something else the author doesn't directly address is that ZFS on Linux is really only usable on 64-bit systems. Funny things may happen if you use the 32-bit version, such as OOPSing when doing simple things such as ls -a.
I've had nothing but great experiences with running this on my home NAS.
On the Freebsd side, there is whole section on tuning [0] for i386 users. Some of it might transpose to linux (at least the concepts and things to watch).
I am terrified by the quality of their non high-end server gear.
It first started with our HP laptop that died multiple times over the warranty period. And then some friend's machines as well .. the central issue in almost all the cases : bad thermal management.
The e-SATA thing probably doesn't matter too much, but for anyone looking to run iSCSI or even higher-throughput NFS, the absence of jumbo frames may be a more important consideration.
These "green" hard drives tend to be very aggressive about parking heads and spinning down the platters. While this is fine if a disk is going to sit idle for a long period of time, in cases like an OS partition and memory buffering, these drives start destroying themselves spinning down and cranking up several times a minute.
We had 4 out of 16 fail one week after burn-in in a raidz configuration. We had plenty of hot spares for various reasons of paranoia, and they didn't go all at once, so we recovered and replaced the entire batch with Constellations.
The only positive note was that we are now firmly in love with zpools and zfs.. It made egregious hardware failure a manageable problem.
Why?
There are a number of the usual installation and adoption issues with FreeBSD and Illumos / OpenIndiana. I love the FreeBSD community, and using dtrace again is a joy. I just don't like having to use them for things I already have solved in Linux or can quickly find documented on either Ubuntu or Arch's wikis.
It's nice after the OS wars to be able to say "well, it's easier for us to use linux because it has a broad user base." :)
Can you please point me to a more authoritative source for this? I also had a "green" drive fail surprisingly soon. How did you confirm that they spin up and down "several times a minute" by themselves. Are you sure it wasn't your misconfigured OS telling them to do that?
Since green drives are also slower, I just assumed they traded off energy for data transfer speed.
It's not transfer speed that they seem to trade off for, it's access times. That may have been part of the problem -- our reads are extremely random over a very broad range of sectors. This isn't exactly a log or video server usage model. :)
The default head parking is 10s. In a ZFS configuration, the other drives time out while waiting for the drive to wake up from a parked configuration. This causes a cascade of IO waves back and forth as the drives in a large array park and then unpark to respond, causing IO to plumment.
Works fine with 1 drive, or even 2 drives. Past that, the probability of a collision and retry on head parking escalates exponentially.
I can't remember the exact specs on our box, but it was a similar microserver that we've got 4 drives RAIDed in. It started at 1tb per drive but we've since upgraded to 2TB drives.
This replaced a FreeNAS setup that I ran out of the closet in my home office for years when I lived in Atlanta. That server was great but was loud, ran so hot the closet was seriously 20 degrees hotter than the office, and an electricity glutton. When we moved to New York City last year, we decided to consolidate to a small unit for size/heat/power.
Highly reccommended to anyone who needs a media server, general file server and fast-access VM/local enviornment.
ZFS is absolutely the way to go.
I just use it for storage though. No VMs etc. And I'm not overly concerned about access speeds so long as I can play HD video off it (which I can).
They may be quiet/inexpensive, but they also don't ship with Time-Limited Error Recovery enabled (http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery). This makes it much more likely they'll drop out of an array. Some models can have this option flipped on; some cannot.
I installed two Caviar Black (b/c of the 5-yr warranty) drives in a RAID array a few months ago. The RAID edition drives aren't even significantly more expensive. I wish I'd known this earlier.
This is a bummer. http://wdc.custhelp.com/app/answers/detail/a_id/1397/p/227,2...
Edit: But on another page, WD says that Caviar drives can be used in consumer RAID configs. Hmm. http://wdc.custhelp.com/app/answers/detail/a_id/996/related/...
This machine has now been chugging along for a long time. It stores about 4 TB of personal backups (all my machines back up to it over the network), and various other things such as projects, media files, photos. ZFS is rock solid. I've had drives fail, and the backup drive take over without noticing a single thing.
I've got 4 GB of memory in this machine and I can get write speeds over the network of 80 MB/sec using consumer grade drives, and read speeds over the network of around 120 MB/sec (I easily saturate my Gbit network).
I wouldn't store my backup bits on any other file system, I've had failures with various Linux based raids/file systems that were nonrecoverable, I've used UFS in the past from FreeBSD and had data be silently corrupted, end to end checksumming is absolutely fantastic!
Initially i tried using Ubuntu Server on it, but there were a few problems with it. I also tried Freenas but beyond the basic "share files and use ZFS" it didn't really offer much in the way of customisation.
So Instead i decided to just put Windows Home Server on it, since all i wanted to do was share files, use basic RAID, and run virtual machines for testing with the minimum of fuss. Windows RDC works fantastically out of the box. For VMs i just used Virtualbox. I stuck Freenas in a VM and left it running for Time Machine - perfect.
I also stuck an SSD i had lying around in the optical bay, so i have 4 drives available in the bays dedicated to storing data.
Despite not using Linux or ZFS, i'm quite happy with this current setup.
This is just wrong. Only last week I resized my RAID5 array from 2 drives (basically RAID1) to 4 drives. Reshaping and resizing the ext4 partition was all done online with minimal performance impact. The only bit that took a long time was the reshaping, and even then it managed to add a new 2TB drive in only a few hours.
Even upgrading the entire array to use a larger size of disk is pretty easy. You do need all the disks to be the same size though :(
Running the FS off a USB stick is a bad idea. No write leveling so its only a question of time before it bombs out.
Plus, sometimes you want logs for troubleshooting... small SSDs are cheap.
On a different note, the number of possible distributions is a tad confusing. Can someone recommend (say FreeNAS or Illumos) the distribution with the most up to date support of ZFS along with ongoing updates? I'm primarily looking for something that is hassle free to maintain and update. FreeBSD or Solaris based is okay.
We looked into using ZFS, as there has been some demand for it, but all the licensing and the people around it were hard to deal with (non-responsive, to be precise). I would be cool, though.
You could have based your product on FreeBSD to use ZFS. There would have been no licensing issue or people to deal with since the BSD license allows modification and redistribution.
Is the HP Microserver the best computer in its class, or are there other good competitors?
GigE can easily be saturated with a server like this.
Also, does anyone know how well dtrace was ported to linux? Last thing I remember reading said it was half-assed
I've been using the native linux kernel driver (obv not in-tree but easy to install) and it's fantastic, I see no slowdowns at all. Whatever performance hit is there if any is worth the benefits :)
... No international shipping? I'm gonna cry.