HGST gets closer to shipping 10TB HDD | Hacker News Reader

HGST gets closer to shipping 10TB HDD

zdnet.com

89 pointstanglesome64 comments

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high.

We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing some pretty sophisticated replication, validation, & caching.

Is there some fundamental obstacle that prevents raid rebuild times from being within a small, constant factor of linear scan time of a single disk?

Naively I would think that it would simply have to stream large chunks of each drive in parallel, apply the parity coding and write out the stream of calculated blocks to the new disk.

Oh sure, it's a big linear streaming read/write - the issue is that if you're taking a 8, 12, 15, etc. hour window to rebuild, there's a fairly high chance of something else going wrong. Particularly because you have to worry that, eg, your flaky power source has been shortening drive lifetimes, that's why you saw the failure within the expected lifetime of the drive, and that's why the another one is going to die in hour 14 of your rebuild.

Of course, that's in the context of an actual RAID array (or ZFS pool, etc.) - if you've got a replication strategy that spreads multiple entire copies over multiple systems, you're less vulnerable and these kinds of disks are a good choice.

Well, that sounds more like worrying about a single point of failure (e.g. that power source that can take out several drives at once) than what the redundancy is designed to protect against, i.e. uncorrelated failures of one of the drives in the array.

Of course having no redundancy at all during a rebuild that takes more than a negligible amount of time is still worrisome, but that's why we have multiple-redundancy raid schemes.

But you still have to untangle your concerns. A single point (or cause) of failure has to be handled differently than increasing probabilities of encountering independent UREs on separate drives during rebuild.

Shingled drives are probably more likely used in a RAID-10 scenario where you're not recalculating parity, but just rebuilding linearly.

These drives (like the Western Digital Purple) are used predominantly for archival data. You should never have to randomly write to, for instance, surveillance video, or similar streamed media.

This isn't for your SQL database server.

This is pedantic, but unless the filesystem is build specifically for WORM devices you always end up with some non-sequential data. In the use cases you're describing they'll be some filesystem metadata (freemaps, directory entries, inode tables) that will need to be rewritten.

Can't this problem be partially solved by having more read/write heads? I don't think it will fit in the same form factor, but if the limit is how fast you can spin the disk then having multiple independently controlled read/write heads per platter seems like one way out of them problem.

seagate dual head design: http://www.tomshardware.com/news/seagate-hdd-harddrive,8279....

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high.

Two disk failures in the same array within 15 hours is very rare.

Unless the root cause is something to do with the array - flakey controller, or power supply.

these kinds of problems may cause the array to fail, but won't necessarily lose data, right? If you have a ZFS pool and a faulty controller causes the pool to fail, ie. "losing" three disks in a raidz2 vdev, my guess is that you would just replace the faulty card and reimport the pool and chances are things would be OK... no?

ZFS can mostly handle that. It will try to recover any data it can, and rebuild whatever it can. Due to how paranoid ZFS is designed, it will survive edge cases like this much better, which is also why I like deploying it here even at home.

And if it can't recover it? You can prove it by failed checksum check. This is why, in my opinion, ZFS is so much better than all other file systems: if it fails, you can prove it instead of just sorta vaguely questioning everything because you can't tell but you suspect it, and it slowly drives you insane.

Once you try ZFS, you never go back.

Had it happen 3 times within a one month window (as in 3x double drive failures in the same array too close together for a rebuild to complete). It may be rare, but manage enough servers and sooner or later these things happens, especially if the cause of the failure is a power supply or the controller.

(We didn't lose any data - everything is replicated elsewhere in addition to the raid)

At scale everything happens, even very rare events.

I have to say this is false. I work in that industry, and even when you have, say, a 4 drive RAID, and all 4 drives are the same model but intentionally different batches, you will still, more often than not, have 2 drive failures within 24 hours.

Now, what does this mean in, exactly? In that four drive scenario, RAID10 has a 1/3rd chance of failure (both failed drives being both the A or B parts of the inner RAID 0s is two out of six possible scenarios), RAID5 will fail no matter which two drives, RAID6 will survive no matter which two drives, ZFS's RAID-Z3 (three parity RAID5/6) will survive no matter which three drives, and RAID1 will survive no matter which three drives.

RAID10 is the most performant of the possible outcomes, and the usual makeup of a 4 drive array unless you absolutely need the storage, then its usually RAID5; unless you seriously do not care about sanity at all, then its RAID0 or just independent disks.

Now, my suggestion for 15 hour rebuild times? Whatever you do, have a hotspare so it can begin immediately rebuilding and do not use any RAID variant that can't handle more than 2 failures in 15 hours. This means no RAID10 or RAID5, only RAID6 or RAID-Z3.

Ceph mitigates this 15 hour time because it can rebuild a lost drive by just allocating new blocks on every other drive simultaneously and maxing out your network and/or storage IO (whichever is slowest) so the window of potential doom is much smaller (depending on Ceph cluster size, obviously, bigger the better in this case).

Frequent enough that I've seen it twice on different arrays. The reason for this is that a drive failure can be caused by an external factor (usually multiple drives fed by the same broken power supply, but not yet so broken that it goes off-line).

I didn't say they don't happen.

They're just so rare that the difference between ~6h rebuilds with 4T spindles versus 15h rebuilds with 10T spindles simply doesn't matter. Any homeopathic advantage of slightly faster rebuilds is dwarfed by the benefits of higher density (less hardware per capacity).

This is true of natural age type failures. When in a RAID every drive is effectively experiencing the same experiences, down to the writes and reads, if the failure is due to the drive's interaction with its environment then one failure almost always predicts more failures.

Natural age failures are pretty rare.

This is probably why people like BackBlaze use RAIN (Redundant Array of Inexpensive Nodes) instead of RAID.

Also, their drive failure is 10 disks/day out of a population of 44,100 drives, about 0.02%.

So, between the lack of correlation of failure (their Nodes, or, "Pods" are in different racks), their ability to lose an entire POD regardless (they have 3 parity pods), and the relatively low disk failure rate - the large drives aren't a problem. A rebuild in 2-3 days is more than fine.

There is a very interesting engineering optimization strategy at play here, would love to do/see a blog post on it.

Simple structures, with a large safety margin and simple control flow vs complex structures, very light weight but at the limits of capacity.

Like an ethernet switch doesn't need complex flow routing algorithms when the raw backplane bandwidth is some multiplier higher than will ever experience contention.

It always seems to me, that things built near their design limits will eventually expose a catastrophic flaw.

Human engineered items that have stood the test of time, Roman Aqueducts, Brooklyn Bridge, DC3, Dodge Dart, Toyota Corolla, AK47, PDP8 all have commonalities in their design philosophies.

> the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high.

Well there's one issue that another whole drive will fail and you're screwed.

The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk.

I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.

That is based off of the BER (bit error rate) of the drive, as stated by the manufacturer. It's usually spec'd at 1e-14 or 1e-15 - that is to say one in every hundred trillion or one in every quadrillion bits. This works out to 12.5TB or 125TB respectively.

http://www.enterprisestorageforum.com/storage-hardware/selec...

The problem is due to a common misconfiguration. Consumer drives have longer recoveries potentially, upwards of minutes, trying to recover a flaky sector. The OS sees this as an unresponsive drive, and eventually does a link reset. This is 30 seconds by default on Linux, not sure about others. The link reset prevents the drive from reporting an explicit read error along with the affected sector LBA. That information is needed for RAID to know what data to rebuild from parity, send that up to the app layer, and also send a good copy back to the sector that reported the read error. So eventually there's an accumulation of these, and in case of a drive failure and another drive that produces a URE, poof, imploded array. Now, you can recover from this, sorta, but it's tedious and requires a sort of skill to do it. So most people give up. Ergo RAID is not a backup. Backup your RAIDs. And make sure drive SCT ERC is shorter than the kernel's SCSI/ATA command timer. Ideally shorten the drive timeout. If that can't be done (consumer drive) then increase the kernel's command timer. Both of these are per device settings.

fun fact: Commodore Amiga floppies used 32bit XOR checksum = 3% of undetected two bit errors!!

http://www.techtravels.org/amiga/amigablog/?p=280

This is probably because Amiga didnt have real hardware floppy-disk controller, just a general IO (CIA) chips, and read raw serial datastream into ram, all the decoding was done in software. Similar to Apple II 2x 8bit XOR checksum.

> Well there's one issue that another whole drive will fail and you're screwed.

You do know that in a RAID6 two drives may fail without causing data loss? In any way you need one or more hotspares ready plugged in in order to keep the time window short.

RAID6 is not perfect but the probability of 3 drives failing during the rebuilt window is much lower than the probability of 2 drives failing. (One also has to consider errors such as memory corruption, chip failures or catastrophic failure to the power supply where no RAID level will protect you from.)

A RAID1 built of two RAID6s may be necessary to avoid performance drops during rebuild. In the case of multiple failures a RAID6+1 setup will protect you from at least 4 hard drives failing in the rebuild time window.

> The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk.

This is another reason for a RAID6. Not only does it recover when one or two disks fail. It also recognizes and recovers broken blocks when one disk returns the wrong data. You scrub the disk weekly and remap broken sectors or swap out (soon to break) harddrives.

Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to rebuild. (Though using erasure coding amplifies the amount of work to be done.)

A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes.

Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?

> Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?

Yes, you create a CRUSH map which lets you define a hierarchical list of bucket types that reflect your failure domains (host, chassis, rack, room, datacenter, etc).

http://ceph.com/docs/master/rados/operations/crush-map/#crus...

Yes, there's no more ticking time bomb so there's no urgency to replace the drive, nor any urgency to max out the drive for 15 hours straight during the rebuild. It can be eased into the cluster over the course of days or a week.

And yes, in the clusters I've built I've always calculated what the chances are that X simultaneous drive failures will take out any data, and it's always been astronomically low.

What's the sweet spot for Ceph cluster redundancy - Original + 3 Copies? I realize the answer is, "It depends on how much redundancy you want" - but I'm guessing that there is a general best practice that Ceph administrators have adopted?

Ceph is a Consistent and Partition tolerant system in the CAP formulation, at the expense of Availability. Fortunately for my use cases I've not needed high availability, which would require two separate clusters and replication between them.

I've mostly seen people use triplicate pools. For semi-warm storage I've been testing erasure coded pools, with a triplicate cache tier on top, and had good experiences on my test cluster.

So, yea, Ceph: https://twitter.com/tbmcmullen/status/566856092185923584

Does this cause problems in practice?

I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.

Ceph isn't ready for production. GlusterFS is, and it also behaves more like a file system, while Ceph is more object storage. They serve different use cases but GlusterFS is more general purpose I think. No MDS required, scales fairly linearly. They're using it at Facebook with XFS and Btrfs underneath depending on workload. At home you can build little ARM gluster clusters, so it kinda even scales down! http://events.linuxfoundation.org/sites/events/files/slides/...

"Ceph isn't ready for production." - this is not the case.

Ceph object-store known as RADOS is production ready and supported by Red Hat [1], in addition the block-device service known as RBD which is an access method to RADOS is also production ready and supported. RBD provides many SAN like features like snapshots, cloning, and soon mirroring. The CephFS (POSIX filesystem access method) is not yet a supported product but is used by several sites in production. Ceph also provides S3/Swift object-store gateway (radosgw) so existing applications can access the object store with a compatible API.

Ceph provides unified storage: providing object, block and filesystem access from a single cluster, so Ceph is the more general purpose technology. Ceph MDS provides metadata service for the CephFS POSIX-filesystem but is not needed if you're not using that feature. For development and testing Ceph will run on a single VM, or more realistically on 3 small VMs so the cluster characteristics can be explored.

[1] alternatively you can buy a production grade fully supported appliance product from Fujitsu based on Ceph known as ETERNUS CD10000. For most deployments Ceph runs well on commodity hardware and the Ceph community provides excellent support too.

> Ceph isn't ready for production

DreamHost would beg to differ. They've been using it for their DreamObject[1] for few years. Granted that they have have better visibility earlier because Sage Weil, but I've seen Ceph deployment for production in the wild since mid-2014.

[1] https://www.dreamhost.com/cloud/storage/

The supplied link suggests it's not production ready, but it's the wrong link. This is the correct link. John Spray, a senior software engineer at Red Hat, "Some people are already using it in production; we're terrified of this. It's really not ready yet." http://www.zdnet.com/article/linux-storage-futures/

EDIT: And I'm not suggestion it isn't badass, just that the idea Ceph as a first suggestion for general purpose RAID replacement is probably not a good idea just yet, and that there's an alternative.

cmurf. What the article and John is talking about is CephFS (which we do use). As you can see that is in the heading of the quote that you took.

What John is not talking about is the Object Store (S3 compatible) or RBD (rados block device, EBS like block storage). There's countless people using both Object Store and RBD in production today. Both of these are the most deployed solutions for OpenStack (for storage or object storage). For example the Bloomberg folks use in production.

We use CephFS, and contributed to it earlier, and helped find a lot of bugs in the kernel and the MDS. Last time we uncounted a bug was over a year ago. We currently only use one MDS with a hot spare (backup node actively replaying the log).

What's really missing today from CephFS is the fsck and restore utilities and I believe people are working on them as we speak.

Clock skew is notoriously difficult to actually fix, especially with a network partition.

I felt weird reading this. HDD are now for backup/capacity mostly, becoming CD-RW/Tapes of the day. Compared to recently shown Apple/Samsung SSDs going above 1GBps [1], feels even more out of place.

[1] http://www.macrumors.com/2015/03/11/13-inch-macbook-air-ssd-...

Apple must not be using the top-tier Samsung M.2 SSD; they can write at 1500MB/s+ and read at 2100MB/s. 1024MB/s is nothing. Now that SSDs are moving to using PCI-E lanes instead of SATA, we'll probably see quite a speed race. Really looking forward to the Intel 750, which will straight up be a PCI-E card for the desktop version.

http://www.tomshardware.com/reviews/samsung-sm951-m.2-pcie-s...

I see a new feature for SATA called Rebuild Assist becoming available to help reduce the RAID recovery times.

http://www.hgst.com/tech/techlib.nsf/techdocs/CCA92CECECFCC7...

The way it works is by realizing there are only a few bad sectors on the failed drive, so rebuild that area first, the used the "bad" drive to help rebuild the rest of the array at full speed.

Sounds clever. Too bad SSD (due to being shit and simply dying all at once) will never support that.

Spinning rust can fail all at once too. When ssds die of write endurance they loudly complain long before they actually lose data.

It can, sure, but I think most failures are of the slower sort.

And SSD's only complain if you listen. Does Windows come with a smart checking tool by default? Don't you have to manually install one and check?

I suspect most people with an SSD have no idea where it's holding.

But it doesn't change the "defective by design" nature. It's one thing if you can't do anything about it - but the bricking design is not good. It should go read only on max writes, not brick.

"And SSD's only complain if you listen." I'm not sure how this is different from HDDs? For both of them you have to listen and install a SMART checking tool. And HDD failure rates are not great either ranging from 2-12% according to recent Backblaze data.

> I'm not sure how this is different from HDDs?

HDDs don't have a set end of life. Instead you use them till the die naturally. SSD you need to be notified when the end of life is.

There is nothing inherently wrong with either way, it's the action AFTER the death (of either) that is the problem: HDDs do they best they can even after a failure. SSDs just die because they are programed to, not because they have to.

Saying: But you are warned does not excuse that.

We stopped using RAID5/6 several years ago because of rebuild time and associated performance drop. Now with these massive disks we don't even use RAID10 any more! Every disk is just paired with one mirror and that's it. (RAID1)

The plan is to move to JBOD soon and just use software to store each file on 3 random disks across 3 separate servers.

That creates a substantial window during which you could lose the second drive.

I'm sure they have their reasons (I won't pretend to know every detail about their setup) but it's not a solution I would adopt in our environment.

RAID 1 rebuild time is the same as RAID 10 but if the entire thing fails you only lose one drive's info, not the entire RAID 10 array.

That seems slow. I have a 6 x 6TB RAID 5 array (27TB) for non-critical data storage and it takes less something like 22 hours to do its weekly parity check. I'm running firmware RAID via an onboard Intel controller on an ASUS board.

Wow, hard drives are getting complicated!

You have to write 256MB to change a single bit?

And you have various zones, and you have to keep track of where data is written, because it's not written in order, and can be on any zone?

You would need some sort of battery backed scratch/cache area to pull this off, so that you don't have to write very often, otherwise I can't imagine performance will be very good.

> You would need some sort of battery backed scratch/cache area to pull this off

Or a copy-on-write filesystem. With some tweaking they can essentially fill one zone by linearly writing new copies (and metadata) to it while garbage-collecting (copying GC, with compactation) old zones in the background

Having maybe 16 to 64 GB of flash storage (for writeback cache) would probably make a significant difference here. I think over the next five to ten years tiered storage will probably go mainstream.

Tiered storage is already here. If you're using block storage in the cloud those guys are already do all sorts of optimizations (for their costs) that include tiering. And on your own desktop / own server side you can do this today on Linux with bcache or dm-cache. Works great for databases.

You only need to rewrite the whole zone if you want to modify something that's before the write pointer. That is, you can write data sequentially without penalty. It's modifying existing data that's the issue.

Before people make too many comments about the complexity and speed of these drives, it is important to realise that these drives are designed for append-mostly archival of data.

Yev from Backblaze here -> gimme gimme gimme gimme!

We spent a bit of time talking about about these "shingled drives" at the LSF/MM summit earlier in the week in Boston. Storage is becoming a more fascinating topic with so many technologies (SSDs, shingled disk, non-volatile memory) and so new open source solutions (Ceph, Luster).

One of the problem with these HDDs is that you still have the same number of spindles. These spindles now need to cover a lot more data, and they were already `challenged`

Yup, RAID is being replaced by object storage for large scale systems. Last week, HGST acquired object storage startup Amplidata...

Reply on news.ycombinator.com