Hacker News Reader

I'd recommend ZFS for working with these kind of arrays; scrub is a command that's easy to regularly schedule, and will only use idle I/O bandwidth (helps that the RAID functionality is integrated with the whole filesystem).

If you have an infrequent scrub policy and hit bad sectors on rebuild, it can detect checksum failures and mark specific files as corrupted, rather than declaring all your disks defective. Traditional linux md raid behaviour is particularly bad in this regard: if you have a raid6 configuration and haven't been scrubbing, and then have a single disk failure, all your disks will have a few random isolated bad sectors (i.e. sectors that will URE when you attempt to read them) on, but since you still have one disk's worth of parity it's possible to recover all your data with no downtime (and with ZFS raidz2 this is what would happen). But with md raid as soon as you hit those bad sectors during the rebuild it will consider those drives as failing and kick them out of the array, and since all your drives have at least one bad sector on that means it's impossible to recover the array.

Well, scrubbing with md is pretty easy, too.

That said, ZFS sounds great, and I've been meaning to learn it sometime. The big thing holding me off has been just -- well, how much time I've put into learning md and such, and how I would find it hard to justify going back to not knowing much (since this is my livelihood).

If you have any tips on "read this, it's a good intro" stuff (slanted to Linux, as that's my cuppa), I'd welcome them.

I'm no expert, just a satisfied user; I got into it from the FreeBSD side, and followed their tutorials, but I don't know how well they'd apply to Linux. In general I like the Gentoo and Arch wikis, particularly for slightly lower-level things like this - even if one is using a different distribution, they tend to take the time and explain the concepts in a way that's mostly distribution-independent.

I will say that ZFS felt very coherent and even - dare I say it - easy. I was worried that merging layers that I'm used to thinking of as separate would mean a loss of control, but I was able to put together all the layouts I wanted. The various commands with a single interface felt a lot like using lvm2 - different tools but under a unifying structure with the same parameters - and a lot of what it does is lvm-like. So my best tip is to approach it more as an LVM that can also do raid-like functionality and be mounted directly, rather than as a raid system that includes volume management.

Sorry if that's not terribly helpful - as I said, I'm not an expert, but I wanted to at least give you some kind of reply.

+1 for the Arch wiki. Despite being a distro wiki they have very nice in depth and distro-independent pages. I use Debian but regularly read through the Arch wiki when trying to do new stuff.

> and will only use idle I/O bandwidth

You should also be doing a nightly SMART disk test, and that too will also only use idle I/O bandwidth.

All my servers do a short test nightly, and a long (i.e. read the entire disk) weekly (during the lowest usage period for each site).

On linux install smartmontools then add

    -s (S/../.././08|L/../../6/00)

to /etc/smartd.conf (adjust the hour and dow for your needs). This will both schedule the tests and keep an eye on them to make sure if there are any errors you will be told about it. (Use the test email command at least once and check!) (Make sure the hour when you schedule the two tests doesn't overlap - if one test is in progress the other won't run, so you may end up never running the long test if you do it wrong - although smartd will tell you about it if that happens.)

Like ZFS, Btrfs will scrub the filesystem on request too. Worth considering.

Just as all hardware and all software RAID solutions as well.

Although, unlike btrfs, ZFS and a lot of other RAID solutions are considered safe for production use.

(while I'm a huge fan of btrfs I've already been waiting many years for it to become stable and the wait is far from over...)

TBH, it's probably good enough at this point in time. ext3 didn't really become 'stable' until Rusty stopped developing it and moved on to ext4 :)

It's a bit different with a file system that still undergoes heavy development of basic features...

http://www.anchor.com.au/blog/2013/04/the-btrfs-backup-exper...

Just don't use btrfs where a failure is going to hurt.

That blog post seems remarkably positive to me: "We thrashed this filesystem and pushed some little used features really hard & found a few showstopper bugs that crop up for our use case. We lost no data & discovered a RAID array error that hadn't been picked up before. In summary, btrfs is great, but we can't use it yet."

Yes, it was better than what I would expect.

But it completely baffles me that anyone saying that that article seems remarkably positive at the same time as saying btrfs is stable...

btrfs is considered safe for production use.

By who?

http://forums.opensuse.org/english/get-technical-help-here/i...

everyone seems to be using this to push their favourite file system, which is fine and all, but if you're using software raid on linux this is the kind of thing you need:

    #!/bin/bash
    #
    # This script checks all RAID devices on the system
    # http://en.gentoo-wiki.com/wiki/Software_RAID_Install#Data_Scrubbing
    for raid in /sys/block/md*/md/sync_action; do
        echo "check" >> ${raid}
    done

(the link referred to seems to be down for me at the moment - i just took this from my main machine).

then add a crontab entry to run it once a week or so:

    0 0 * * 0 /root/bin/scrub-raid.sh &

you can check the script by running it by hand and then doing

    cat /proc/mdstat

where you'll see something like:

    Personalities : [raid1] [raid0] [raid10] [raid6] [raid5] [raid4] [linear]
    md1 : active raid1 sdb1[0] sdc1[1]
          976760640 blocks super 1.0 [2/2] [UU]
          [=>...................]  check =  6.0% (59419520/976760640) finish=191.3min speed=79902K/sec
          bitmap: 0/8 pages [0KB], 65536KB chunk

finally, my notes on this - http://www.acooke.org/cute/ScrubbingR0.html

They didn't mention RAID, and they talk about using SSDs. The failure mode you describe (timeouts) is one typical for spinning rust drives, and not for SSDs.

Typical for a killer backplane / killer power supply.

Feeding 15 or so volts out the 12 volt line usually has negative results after a couple hours

I went thru something like this circa 1996. It was not fun at all.

Sometime similar happened with some networking gear at a previous employer. "I know we have pri/sec control cpu boards, but they're burning out as fast as I can swap a new one in!"

"we obviously store data in mirrored mode on several servers"

They then asked the provider to swap the disk. They had a RAID1; it's a common enough setup.

SSDs fail in many of the the same ways as spinning disks, they just add a few more failure cases.

Scrubbing should be done regularly (weekly?)

I don't think this applies here. They just moved over to their new RAID setup last Saturday, so they had a major failure within about 72 hours of that transition.

They use a managed hosting provider, which means that there is no guarantee that the disks are new. When one customer finishes with disks, they get wiped (hopefully) and passed on to the next customer.

Even if they are new, though, you never know if it's completely good until you've tested every block on the device. These days most format operations don't actually wipe out the whole disk (takes ages!) so even formatting on a brand new disk won't tell you if it's valid.

That said, the odds of having multiple disk failures on a brand new RAID in 72 hours are pretty low. If that's the case, I'd start to suspect a bad batch of disks and/or bad backplane or other hardware (bad power?) that is causing the hardware to fail.

Note that they did periodic backups, which would have scrubbed the same sectors as a full resync, so that's not the case.

Not really.

A backup would be a filesystem level event, so at best it would only access 1/N of the sectors (where N is the duplication level). It isn't guaranteed to access all of the sectors the data resides in -- and also, it may have hit the kernel's filesystem cache or the on-disk cache or something like that, too, bypassing the physical media entirely.

Also keep in mind that a RAID rebuild is a media level event, it doesn't just copy files. It will copy over the "empty" sectors, too, because it just has no idea what's what on the disk. It faithfully recreates the raw bytes on the media.

The disk might have failed in a part of the disk you aren't actually using and never noticed because all of your actual data is intact -- the the RAID can't really rebuild without them.