Hacker News Reader

Comment by dragontamer | Hacker News Reader

I have to imagine that by making Storage Pods 1.0 through 6.0, maybe they "encouraged" Dell (and other manufacturers) that this particular 60+ hard drive server was a good idea.

And now that multiple "storage pod-like" systems exist in the marketplace (not just Dell, but also Supermicro) selling 60-bay or 90-bay 3.5" Hard drive storage servers in 4U rack form factors, there's not much reason to build their own?

At least, that's my assumption. After all, if the server chassis is a commodity now (and it absolutely is), no point making custom small runs to make a hypothetical Storage Pod 7. Economies of scale is too big a benefit (worst case scenario: its now Dell's or Supermicro's problem rather than Backblaze's).

EDIT: I admit that I don't really work in IT, I'm just a programmer. So I don't really know how popular 4U / ~60 HDD servers were before Backblaze Storage Pod 1.0

View on HN

3U and 4U x86 whitebox servers designed for any standard 12"x13" motherboard, where the entire front panel was hotswap 3.5" HDD bays were already a thing many, many years before backblaze existed.

What wasn't really a thing was servers with hotswap HDD trays on both ends (like the supermicros) and things that were designed with vertical hard drives dropped down from a top-opening lid to achieve even higher density.

The Sun X4500 "thumper" server had 48 drives, if I remember correctly, and came out in 2006ish.

It had hot-swap SATA disks (up to 512GB disks initially!) and was actually pretty cool and forward thinking

https://web.archive.org/web/20061128164442/http://www.sun.co...

IIRC the company I was at at the time had a set of thumpers for something. Maybe a SAN?

They were incredibly cool for the time.

I remember when I worked in a company that was a netapp shop, and in 2007, they sent one of the ZFS developers and andy bechtolsheim to sell us thumper and ZFS. The developer and I had a spirited discussion of the relative merits of having volume management built into the filesystem.

At the time I advised my company to skip the storage servers (which were overdesigned IMHO, but were unquestionably stout hardware) but that ZFS was an interesting filesystem to explore.

Hah! That was the system my small university IT department bought to act as a storage server. Actually they bought 2, with one to act as a backup.

Unfortunately for them, no one knew how to administrate Solaris, so they... installed Windows. No idea if they actually succeeded in doing anything useful with the systems.

Bought three with 1TB drives and ZFS. Wasn’t cheap.

Hardware was great, but no linux/zfs support meant we didn’t stick with it.

A couple of years later we had standardised on supermicros with a similar density and ran linux on them.

Sun lost a fair amount of money from us (7 figures) by choosing not to embrace linux.

That's not the backblaze design. The backblaze design is that the drives are individually hot-swappable without a tray. 60 commodity SATA drives that can be removed and serviced individually while a 4U server continues to operate normally is pretty amazing.

that's what I said, the backblaze design is:

> things that were designed with vertical hard drives dropped down from a top-opening lid to achieve even higher density.

the backblaze storage pod system and some 3rd party derived designs have the SATA/power ports on PCBs facing upwards, mounted at bottom interior of rack chassis, and 3.5" HDDs mounted vertically down into the system.

The company which manufactures the metal cases created a spinoff company called 45Drives which sells commercially supported pods.

They are fantastic.

You'd think they could build a ARM-powered, credit card sized controller for them with a disk breakout card and network IO. PC motherboard and full-sized cards seem like overkill.

They're running a fair bit of math (probably Reed Solomon matrix multiplications for error correction) over all the data accesses.

Given the bandwidth of 60+ hard drives (150MB/s per hard drive x 60 == 9GB/s in/out), I'm pretty sure you need a decent CPU just to handle the PCIe traffic. At least PCIe 3.0 x16, just for the hard drives. And then another x16 for network connections (multiple PHY for Fiber in/out that can handle that 9GB/s to a variety of switches).

We're looking at PCIe 3.0 x32 just for HDDs and Networking. Throw down a NVMe-cache or other stuff and I'm not seeing any kind of small system working out here.

---------

Then the math comes in: matrix multiplications over every bit of data to verify checksums and reed-solomon error correction starts to get expensive. Maybe if you had an FPGA or some kind of specialist DSP (lol GPUs maybe, since they're good at matrix multiplication), you can handle the bandwidth. But it seems nontrivial to me.

Server CPU seems to be a cheap and simple answer: get the large number of PCIe I/O lanes plus a beefy CPU to handle the calculations. Maybe a cheap CPU with many I/O lanes going to a GPU / FPGA / ASIC for the error checking math, but... specialized chips cost money. I don't think a cheap low-power CPU would be powerful enough to perform real-time error correction calculations over 9GBps of data.

--------

We can leave Backblaze's workload and think about typical SAN or NAS workloads too. More I/O is needed if you add NVMe storage to cache hard drive reads/writes, tons of RAM is needed if you plan to dedup.

Xeon E5-1620 last I saw

I'm not familiar with the algorithms, but matrix multiplication sounds well suited towards GPUs. I wonder if you could get away with a much cheaper CPU and a cheaper GPU for less cost?

But the main issue with GPUs (or FPGAs / ASICs) is now you need to send 9GBps to some other chip AND back again.

Which means 9GBps downstream (to be processed by the GPU) + 9GBps upstream (GPU is done with the data), or a total bandwidth of 18GBps aggregate to the GPU / FPGA / ASIC / whatever coprocessor you're using.

So that's what? Another 32x lanes of PCIe 3.0? Maybe a 16x PCIE 4.0 GPU can handle that kind of I/O... but you can see that moving all this data around is non-trivial, even if we assume the math is instantaneous.

---------

Practically speaking, it seems like any CPU with enough PCIe bandwidth to handle this traffic is a CPU beefy enough to seemingly run the math.

PCIE 3.0 is 1GB/s per lane in each direction. A 3.0 8x link would do a good job of saturating the drives. And basically any CPU could run 8x to the storage controllers and 8x to a GPU. Get any Ryzen chip and you can run 4 lanes directly to a network card too.

If only the ASIC on the HDD could run these computations and correct bit errors right during data transfers!

The HDD ASIC certainly is doing those computations.

The issue is that Backblaze has a 2nd layer of error correction codes. This 2nd layer of error correction codes needs to be calculated somewhere. If enough errors come from a drive, the administrators take down the box and replace the hard-drives and resilver the data.

Backblaze physically distributes the data over 20 separate computers in 20 separate racks. Some computer needs to run the math to "Combine" the data (error correction + checksums and all) back into the original data on every single read. So singular hard drive can do this math because the data has been reliably dispersed to so many different computers.

> They're running a fair bit of math (probably Reed Solomon matrix multiplications for error correction) over all the data accesses.

Do those run on this machine? I imagine backblaze has redundancy at the cluster level rather than machine level. That allows them to lose a single machine without any data becoming unavailable. It also means we shouldn't assume the erasure code calculations happen on a machine with 60 drives attached. That's still possible but alternatively the client [1] could do those calculations and the drive machines could simply handle read/write raw chunks. This can mean less network bandwidth [2] and better load balancing (heavier calculations done further from the stateful component).

[1] Meaning a machine handling a user-facing request or re-replication after drive/machine loss.

[2] Assume data is divided into slices that are reconstructed from N/M chunks, such that each chunk is smaller than its slice. [3] On read, the client-side erasure code design means N chunk transfers from drive machine to client. If instead the client queries one of the relevant drive machines, that machine has to receive N-1 chunks from the others and send back a full slice. (Similar for writes.) More network traffic on the drive machine and across the network in total, less on the client.

[3] This assumption might not make sense if they care more about minimizing seeks on read than minimizing bytes stored. Then they might have at least one full copy that doesn't require accessing the others.

> Given the bandwidth of 60+ hard drives (150MB/s per hard drive x 60 == 9GB/s in/out)

Given their scale and goal, it would be pretty wasteful to build it to max the writing speed of all hard drives. Considering you rarely write on the pod, you would be better of getting a fraction of that speed and writing on multiple pods at the same time to get the required peak performance.

In fact actually that makes much more sense to put that math on some ingests server and theses hard drive servers would simply write the resulting data. It makes it much easier and faster to divide it over 20 pods like they currently do.

Definitely limited by the 1gbps or even 10gbs network connection

Any pod like this would normally have at least 1x40gbps uplink minimum?

Like most blade setups, like (random example): https://www.storagereview.com/review/supermicro-x11-microbla...

Storage Pod 6.0 seems to be 2x10Gbps Ethernet: https://www.backblaze.com/blog/open-source-data-storage-serv...

I don't think that makes sense for a backup application.

Read load alone can be pretty high.

And no, you want to calculate checksums and fix bit errors right here in the RAM buffers you just read or received, because at such scales hardware is not error-free.

> Considering you rarely write on the pod

Isn't Backblaze's workload nearly exclusively writes? People back up far, far more often than they restore.

I think from the other commenters perspective the workload is almost exclusively idle. Also they do object storage now, not just backup

That's not really how it's done.

RS is normally used as erasure code: It's used when writing (to compute code blocks), and when reading _only when data is missing_. Checksums are used to detect corrupt data, which is then treated as missing and RS used to reconstruct it. Using RS to detect/correct corrupt data is very inefficient.

Checksums are also normally free (CRC + memcpy on most modern CPUs runs in the same time that memcpy does: it's entirely memory bound).

The generation of code blocks is also fairly cheap: Certainly no large matrix multiplications! This is because the erasure code generally only spans a small number of blocks (e.g. 10 data blocks), so every code byte is only dependent on 10 data bytes. The math for this is reasonably simple, and further simplified with some reasonable sized look-up tables.

That's not to say that there is no CPU needed, but it's really not all that much, certainly nothing that needs acceleration support.

They already have a RAID card doing it for them

A French SDS company, now owned by OVH, did exactly that a few years ago : https://fr.slideshare.net/openio/openio-serverless-storage-7...

I don't think they actually sold any.

I have seen some people build Ceph clusters using the HC2 board [1] before. I'm not sure what the performance is like, but it seems like a neat way to scale out storage. The only real shortcoming is that there's a single NIC... If there were two, you could use an HA stack for your network and have a very robust system for very cheap.

[1] https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/

With 60+ hard drives in a single enclosure, you want all of the processing power, PCIe lanes, and bandwidth you can get.

There are a small number of high end ARM server boards that could do it, but you’re not saving much money at that point. Might be more expensive due to lack of scale.

Amortize a server-grade CPU and motherboard across 60+ high capacity drives and it’s not really worth pinching pennies at the risk of lower performance.

They existed for other applications, like NetApp and the ZFS Appliance. Long, long before 2009.

Reply on news.ycombinator.com