“When all your engines stop, the flight is over”
blog.theoldreader.com
blog.theoldreader.com
In case it's helpful, and for general knowledge dissemination:
What likely happened is that a drive "failed". This is usually when the RAID card decides that a drive has had enough command errors that it fails the drive. It may actually be fine, and just had a spate of bad responses. You might try to online the drive again and let it rebuild, but that's debatable.
At any rate, so they replaced the drive. That's fine. But then, to rebuild the RAID back to optimal state, it has to read all of the data off of the other drives. Here's where a bad scrubbing policy bites you -- because if those drives have any sectors that have gone bad or problems with the hardware, those drives might fail as soon as the rebuild runs.
Scrubbing should be done regularly (weekly?). What it does is, in essence, test every sector on all of the disks in the array to make sure that all of them are still fully functional so that -- if there is a failure -- you're pretty sure you can rebuild.
The downside of scrubbing is that, for better or worse, it does exercise your disks fairly heavily. Also, if you don't have a suitable trough period then you might even find it difficult to have the available I/O bandwidth to do it.
That said, you should do it if you're not.
A backup would be a filesystem level event, so at best it would only access 1/N of the sectors (where N is the duplication level). It isn't guaranteed to access all of the sectors the data resides in -- and also, it may have hit the kernel's filesystem cache or the on-disk cache or something like that, too, bypassing the physical media entirely.
Also keep in mind that a RAID rebuild is a media level event, it doesn't just copy files. It will copy over the "empty" sectors, too, because it just has no idea what's what on the disk. It faithfully recreates the raw bytes on the media.
The disk might have failed in a part of the disk you aren't actually using and never noticed because all of your actual data is intact -- the the RAID can't really rebuild without them.
If you have an infrequent scrub policy and hit bad sectors on rebuild, it can detect checksum failures and mark specific files as corrupted, rather than declaring all your disks defective. Traditional linux md raid behaviour is particularly bad in this regard: if you have a raid6 configuration and haven't been scrubbing, and then have a single disk failure, all your disks will have a few random isolated bad sectors (i.e. sectors that will URE when you attempt to read them) on, but since you still have one disk's worth of parity it's possible to recover all your data with no downtime (and with ZFS raidz2 this is what would happen). But with md raid as soon as you hit those bad sectors during the rebuild it will consider those drives as failing and kick them out of the array, and since all your drives have at least one bad sector on that means it's impossible to recover the array.
That said, ZFS sounds great, and I've been meaning to learn it sometime. The big thing holding me off has been just -- well, how much time I've put into learning md and such, and how I would find it hard to justify going back to not knowing much (since this is my livelihood).
If you have any tips on "read this, it's a good intro" stuff (slanted to Linux, as that's my cuppa), I'd welcome them.
I will say that ZFS felt very coherent and even - dare I say it - easy. I was worried that merging layers that I'm used to thinking of as separate would mean a loss of control, but I was able to put together all the layouts I wanted. The various commands with a single interface felt a lot like using lvm2 - different tools but under a unifying structure with the same parameters - and a lot of what it does is lvm-like. So my best tip is to approach it more as an LVM that can also do raid-like functionality and be mounted directly, rather than as a raid system that includes volume management.
Sorry if that's not terribly helpful - as I said, I'm not an expert, but I wanted to at least give you some kind of reply.
You should also be doing a nightly SMART disk test, and that too will also only use idle I/O bandwidth.
All my servers do a short test nightly, and a long (i.e. read the entire disk) weekly (during the lowest usage period for each site).
On linux install smartmontools then add
-s (S/../.././08|L/../../6/00)
Although, unlike btrfs, ZFS and a lot of other RAID solutions are considered safe for production use.
(while I'm a huge fan of btrfs I've already been waiting many years for it to become stable and the wait is far from over...)
http://www.anchor.com.au/blog/2013/04/the-btrfs-backup-exper...
Just don't use btrfs where a failure is going to hurt.
But it completely baffles me that anyone saying that that article seems remarkably positive at the same time as saying btrfs is stable...
Feeding 15 or so volts out the 12 volt line usually has negative results after a couple hours
I went thru something like this circa 1996. It was not fun at all.
Sometime similar happened with some networking gear at a previous employer. "I know we have pri/sec control cpu boards, but they're burning out as fast as I can swap a new one in!"
They then asked the provider to swap the disk. They had a RAID1; it's a common enough setup.
SSDs fail in many of the the same ways as spinning disks, they just add a few more failure cases.
#!/bin/bash
#
# This script checks all RAID devices on the system
# http://en.gentoo-wiki.com/wiki/Software_RAID_Install#Data_Scrubbing
for raid in /sys/block/md*/md/sync_action; do
echo "check" >> ${raid}
done
then add a crontab entry to run it once a week or so:
0 0 * * 0 /root/bin/scrub-raid.sh &
cat /proc/mdstat
Personalities : [raid1] [raid0] [raid10] [raid6] [raid5] [raid4] [linear]
md1 : active raid1 sdb1[0] sdc1[1]
976760640 blocks super 1.0 [2/2] [UU]
[=>...................] check = 6.0% (59419520/976760640) finish=191.3min speed=79902K/sec
bitmap: 0/8 pages [0KB], 65536KB chunk
I don't think this applies here. They just moved over to their new RAID setup last Saturday, so they had a major failure within about 72 hours of that transition.
Even if they are new, though, you never know if it's completely good until you've tested every block on the device. These days most format operations don't actually wipe out the whole disk (takes ages!) so even formatting on a brand new disk won't tell you if it's valid.
That said, the odds of having multiple disk failures on a brand new RAID in 72 hours are pretty low. If that's the case, I'd start to suspect a bad batch of disks and/or bad backplane or other hardware (bad power?) that is causing the hardware to fail.
"But even today a 7 drive RAID 5 with 1 TB disks has a 50% chance of a rebuild failure. RAID 5 is reaching the end of its useful life. "
http://www.zdnet.com/blog/storage/why-raid-5-stops-working-i...
They are too relatively slow for home computer use but for servers, much more reliable.
That said, this chart concerns me:
http://www.ssdaddict.com/ss/Endurance_cr_20130122.png
25nm Vs 34nm http://google.com/search?q=cache%3Ahttp%3A%2F%2Fwww.xtremesy...
My first personal computer SSD is going to be the Samsung 830 from different batches in Raid1
Also, when someone else builds your servers, you should query the smart info from the drive to make sure they aren't used SSD.
Not even Intel SSDs are immune: One of the Debian developers has reported that the SSDs shipped in the latest Thinkpads die if you try and construct an encrypted filesystem on them. Somehow they corrupt themselves during the initial write of random data to the disk.
(Interesting that these SSDs died whilst under high write load too: is this a particular weak point for some reason?)
You should not trust anything for "reliable storage". That's what backups and redundant drives are for.
“When all your engines stop, the flight is just starting"
Sadly, the number of open fields to aim for in the SF Bay Area is pretty small. It's either hillsides, bay, or city. Oh, and a couple of thin dikes you could land on. ("That's why nailing the centerline is so important. Someday you might need to land on something like that dike wall -- and if you nail it, you walk away fine.")
Different job - had a developer accidentally run a where-clause-less delete in production. Same net result. RAID and SANs are definitely not backup solutions.
When your engines stop and you're getting lots of eyeballs, don't assume they all know you.
It's still unclear how the storage failure was related to the migration. Was the new engine/fs disruptive to the SSD?
Edit re migration connection: Anything that stresses hardware already close to the edge can trigger a failure.
Pedantic and off-topic as it is, this is incorrect. At least for airplanes.
When all your engines stop you continue to glide, and you can even manage to land succesfully with a little skill and luck.
Similarly just because a rebuild fails doesn't mean that the data is lost. Just pop in the drives individually and fetch whatever you can from all of them, it is seldom the case that the exact same parts of them become irretrievable at the same time or that the drives stop function completely. It's just some more work than a regular rebuild.
By the very definition of the world "flight", yes.
A big mistake to make is to fill an SSD drive completely up with data. This is a huge no-no unless it's an enterprise drive which usually use a totally different design.