Backblaze's Hard Drive Stats for Q1 2018

> Backblaze's APIs are not S3 compatible. .... I'll never understand the design choices behind their API

It is all clear if you understand our APIs are cheaper to implement than Amazon S3's APIs. I freely admit the APIs are LITTLE BIT more complicated than S3 (which is not a good thing). Let me start from the beginning and explain why:

We originally never planned to open the APIs up for general use. We created the Backblaze Personal Backup client and we owned both the client and the server, so we were originally willing to do a tiny bit more programming to get rid of some expensive equipment and choke points in the datacenter.

In the S3's APIs, you simply "upload" and you are done. This means there is a very high availability choke point on upload. With Backblaze client (and later "B2" APIs), there are no choke points. The Backblaze client contacts (HTTPS) a central dispatch server and asks the dispatch server for a pod (computer) that contains some spare disk space. Then something very very important occurs that does not occur with Amazon S3 -> the client entirely stops talking with the central dispatch server. And then the client contacts the pod DIRECTLY to perform the uploads.

Now the "contract" with the storage pods is that if that one storage pod crashes, or fills up, or decides not to talk with the client anymore, the client is absolutely responsible to go back to the "dispatch server" and request a new pod somewhere else in the Backblaze datacenter to upload data into.

By requiring these one or two extra steps, Backblaze does not have to buy any expensive load balancing hardware, and we only need 10 Gbit network cards, there are no central choke points requiring multi-Terabit networking speeds like Amazon. Instead we have several thousand individual 10 Gbit network cards which cost us almost nothing because they come built into the Intel motherboards nowadays. The switches cost a few bucks per port.

I assume that Amazon S3 accepts the data, then has to "move it" to the final destination with a network copy. In Backblaze, the data literally lands in the final destination coming straight from the client. Fewer copies means lower cost and higher performance.

Backblaze purchases no load balancers, except for some "tricks" to make the "dispatch servers" highly redundant. But the dispatch servers have only TINY trickles of info coming and going, and the clients only have to contact the dispatch server ONCE every few days (then upload endlessly to the final destination pods), so even the dispatch servers can be "only" 10 Gbit.

> Backblaze is incompatible with Amazon S3

TL;DR - by requiring clients to do a small amount of extra programming and making programmers understand the "contract" (which is the client must retry some things and handle a few more errors), Backblaze cuts out a lot of cost out of our datacenter while INCREASING scalability over S3. We pass the money savings along to our customers. We throw in the increase scalability over S3 for free. :-)

Loosely speaking files are encoded in to to chunks and scattered across multiple disks (actual stored data is more than the size of the original object, but less than full duplication, triplication or other redundant ways to store it).

That blog suggests Backblaze use a Reed-Solomon ratio of 17:20 ratio, i.e. every object is encoded in to 20 chunks, for which you only need to retrieve 17 different chunks to be able to decode and get the full object.

If they're putting only one chunk per drive, that gives any object stored resilience from up to 3 disks failing, at a storage cost of only 1.17x original object size (20/17 ~= 1.176). They'll likely have repair processes to proactively handle any loss of shards from a disk failure, too.

Reed-Solomon is giving them something akin to triplication, while using significantly less disk space than duplicating.

Nothing comes for free, though. The additional cost here is the overhead of encoding and decoding the object, and on top of that the co-ordination etc required from requesting chunks from multiple locations. That adds some unavoidable latency, but that article suggest Backblaze find the JVM to be extremely fast (as fast as a pure C implementation of Reed-Solomon).