Comment by brianwski | Hacker News Reader

Comment by brianwski | Hacker News Reader

Parent Full thread

At the lowest level there are three RAID groups in each pod. Each RAID group is made of 15 drives configured in software RAID 6 with 2 parity drives. This means you can lose 2 dives and the data is entirely safe and intact. If 3 or more drives completely fail simultaneously (not just pop out of the RAID group or power down, but where that drive is lost forever, like it will never power up again) you will lose at least some of the data on that RAID group. Layered on top of the 15 drive RAID group is LVM. The specifics of the LVM config are there is one PV (Physical Volume) spanning the 15 drives, then on top of that are one VG (Volume Group) spanning the same exact 15 drives. Then on top of that are as many LV (Logical Volumes) as it takes to keep each logical volume under the ext4 limit of 16 TB. With 3 TB Hitachi drives, there are 3 separate LV on top of the same exact 15 drives. Finally, there is one ext4 file system sitting on top of each of the LV (one to one with the LV). Disclaimer: I work at Backblaze, but datacenter and pods aren't my main area of focus.

While you are here....Why only a single boot drive? It seems like an obvious failure point, and at only $40 per drive I would think soft raid 1 would be no-brainer for reliability.

Regardless though, super impressed by the work into rolling your own hardware, hope you guys continue to do well.

I just asked, and found out we actually HAVE had a number of boot drives fail in our fleet of 200 pods. Most decisions in the pod are around saving money, so our initial thoughts were just that no customer data is on the boot drive so it isn't all that important. But don't get me wrong, there are SO MANY GOOD opportunities to improve the pod, Backblaze just stops working on the pod when it does what is needed for us and we run off to focus on other things. Your call on the boot drive is every bit as valid as ours. :-) I'm staring at an open pod here and I see plenty of good spots to put a second boot drive, and we'll probably be going to a 2.5" form factor (laptop) boot drive sometime soon which would yield even more space.

(Just FYI, I'm sure you guys have already thought of or experimented with this...)

We've had good luck so far with using small USB flash drives for booting big file servers. We keep the drive image pretty generic and if there's a problem with one, we just replace it with a cloned USB flash drive and reboot, no problem.

It doesn't seem to hurt performance at all for these kinds of uses -- although we do set it up without swap to keep the life of the USB flash device reasonable, which might or might not work in your case.

Personally, I'd eliminate the boot drive and PXE boot.

We're considering a PXE boot solution (among other solutions) just to keep all 200 pods (and growing!) updated to the latest Debian. But we also use the boot drive for a few other things like error logging and such. But the idea of eliminating the boot drive entirely is not a bad idea, we could drop the logs in a folder at the top of the data drives. We already (selectively) mirror various excerpts of the logs to other machines in case the whole pod disappears from the network so we have some history and understanding of what was going on when it went missing.

Perhaps streaming logs would help you out? My employer dumps a preposterous amount of log data constantly via streaming log systems (we use log4j, but there are solutions for syslogs, etc.). Aside from a few early hiccups, it works pretty well.

Just curious: Did you consider XFS? It would avoid the 16TB limit with ext4.

Thanks!

Reply on news.ycombinator.com