Backblaze Hard Drive Stats Q2 2020

Crypt will encrypt all files before they leave your device, yes.

Is here a reason they don't just supply you with 5he encrypted data and give access to a tool to decrypt it?

This is the only thing putting me off backblaze

That is strange. The encryption, in that case, only really offers protection against data breaches

> Today reliability is sufficient that one can meet a given data availability goal by replicating the data 2, 3, 4, 5, <whatever> times as there is only once in a blue moon a bad batch of drives when they tried out a new bearing lubricant or so.

Unless you have an uptime bug in the firmware where all your drives die at once:

* https://www.zdnet.com/article/hpe-tells-users-to-patch-ssds-...

It's a factor of how quickly they can replace drives and how well redundant data is spread between disparate systems. IIRC, they make sure data is dispersed not only at the chunk and drive level, but the system and rack level (and maybe datacenter level? not sure).

At that point, if there's not contingency redundancy built in (See below), it's really a matter of how long it takes to replace a drive (in both identifying the problem, physically replacing the hardware, and replicating data to it). There's a lot of (fairly simple) math involved in running down those numbers, but based on the percentage of drives that fail in a quarter, I think it would take both a spectacular run of bad luck combined with negligence on their part in making sure redundancy levels are kept over a longer period to actually have problems.

> Is there not the danger that the quality drops drastically to the point that one would need an unreasonable number of copies?

I think the very simple way to look at this is that space capacity and automatic redundancy checking can account for a lot of bad drives. E.g. if a drive has 100 chunks of data all copied to 100-200 other drives and systems (such that there are three copies of any chunk), that the data exists three places, and if that drive dies and the system detects those 100 chunks are now only exist in two places, it can immediately locate 100 locations that have capacity to receive a chunk and start replicating data to keep the level of redundancy they need. Even if there was a very large set of bad drives, they would have to all go bad in a very short time frame, short enough that the couldn't be physically swapped out and data couldn't be copied across the network, for it to cause a problem.

At least that's how a system like this could be developed, and my understanding is that Backblaze's system works like this to some degree.

> (HP logic analyzers come to mind, but also Boing airplanes)

Typo of the year, "Boing 737MAX" sounds more like a basketball than something I'd want to fly in.

If you use software-raid you can use striping.

Basically, divide up each disk into 1TB partitions, then RAID the 1TB partitions together. Synology uses this for their Hybrid RAID. Obviously care needs to be taken to not confuse partitions but this lets you use disks of differing sizes.

Unraid also offers RAID with differing sizes for disks, it has quite different failure modes though but it's been fairly reliable for me.

You don't need hardware raid controllers. (Unless your really need them for some reason) Software RAID is perfectly fine for most use cases.

Affordable disk shelves - something like the Netapp DS4243 - on the other hand tend to be reliable and give plenty of space for SAS and SATA drives. I bought a fully populated shelf (4 x powersupply (of which I'm only using 2, the rest are disconnected and meant as spares), 2 controllers and 24 15K SAS drives (mostly HGST, a few Seagate drives) for around €400. I'm currently running it with a number of those 15K drives but will eventually replace some of them with slower but bigger SAS or SATA drives, not so much to get more space but to save power. For now the thing runs fine, the error logs on the drives are mostly clean (2 drives with a few errors, mostly likely due to power glitches as they occurred a long time ago and have not recurred). If you want lots of space for drives with redundant power and cooling these shelves are a good value since there are plenty of them on the market which keeps prices down.

I'm using a combination of striping, RAID and mirroring depending on the content, using lvm and mdadm instead of ZFS for the increased flexibility. The server itself contains a HP R410i RAID card which drives 8 internal 2.5" SAS drives, the thing could be expanded for not that much money to a 16 x 2.5" array but I see no reason to do so given the higher prices for 2.5" drives and the abundance of capacity offered by the DS4243.

An mdadm is fine too.

Here's the part list: https://uk.pcpartpicker.com/list/fcxgk6

I shopped around and waited for sales/price drops and got it all for under £500 (not including drives). I don't the pricing if your're not in the UK, so it may be cheaper elsewhere to do another combination.

As for drives wait for good deals on WD Elements/My Book Desktop External Hard Drive and shuck them. Note, only do this for +8TB drives (these are CMR drives atm) as under that size the drives are SMR which you should avoid like the plague for RAID/ZFS as they have been reported to fail if you ever need to rebuild. So then you're looking at 4x£120 which is another ~£500ish investment.

If you want to save money get a preowned box like the N54L off eBay at ~£100, and spend the money on drives, then stick FreeNas on it and if you're only file-serving the box is good enough.

I think the custom box is a bit overkill but it's got so much room for upgrades, I plan on getting 8x10TB drives in there one day, plus you can run VMs on it as the i3 processor isn't too bad.

FWIW I out together a ragtag NAS (recycling CPU and PSU from elsewhere) for around $400. Even if you start from scratch, the main cost is storage. If you're not afraid to void some warranties, you can get 8TB drives for around 30% less by shuccing WD external drives

FWIW I wonder if they're still run as something of a separate group, but given that the article says the HGST brand was removed in 2018, who knows what speed the long term merge might occur upon.

Explains why HGST brand drives are harder to come by I think.

Do you power cycle the disks during regular operations?

There is a failure mode in disks which can be modelled as something fails on the drive, but it continues to work fine for maybe months or years until the next power cycle, upon which it then won't work.

Obviously that's a problem for redundancy schemes because you think you have plenty of redundancy till there is a power outage and suddenly loads of drives fail at once.

I have never seen any of your reports measuring or reporting on these 'fail after power cycle' events, which is surprising.

I recall reading about your triple redundancy with R-S codes, but it's good to restate it for each audience.

From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents, and it feels like a waste. But it's possible that my comfort levels for available disk space are a lot more conservative than other people's, and so the difference is less pronounced in a 'normal' storage situation.

For instance, an array that's at 80% capacity takes 25% longer to rebuild than I wish it would, whereas an array that's at 66% capacity takes 50% longer.

I'm interested that you don't use nested codes...

You could then for example have 18/2, and then group together 400 drives in a 2nd layer of 19/1. Hey, I reckon you could do 19/1,39/1, reducing your storage costs 7.5%...

Sure, the worst case rebuild cost is much worse, but overall data loss probability is far lower, and a 2nd layer rebuild is a very rare event, and in that case, your customers totally prefer a few extra seconds latency over an email reporting their data is lost...

I assume you mostly do streaming rather than random writes, so the overhead is evenly spread amongst the disks, and is the same 15% as your current scheme.

Recertified/Refurbished drives are never worth it IMHO.