Raspberry Pi based Ceph Storage Cluster with 15 nodes and 55 TiB raw capacity | Hacker News Reader

Raspberry Pi based Ceph Storage Cluster with 15 nodes and 55 TiB raw capacity

hackaday.io

40 pointslouwrentius7 comments

Nice, so this setup requires 18 raspberries for 30 USB-3 hard disks, 2TB each. I know this is about hacking and fun, but I wonder if there's no more better, certainly more traditional way of attaching these external disks to some powerful computer. What about a single server and attaching them with USB-3? Power consumption will be comparable with idling processors, but I assume the overall performance will be better with a traditional x86 desktop or server processor.

I've considered such a setup myself and decided that I also wanted redundancy at the hardware/node level. If my storage server goes down, then everything is down until it's fixed. I think the person who made this setup had the same considerations

It's an interesting approach and I think it might be reasonable in terms of power. I also really like the amount of redundancy in the setup, but the cost is probably too steep for my preferences

An RPI 4 8gb is approximately 90$, so 15 nodes becomes 1350$

I think there's room for a board with a reasonable sized (16gb-32gb) emmc, 2-4 sata ports (with 4-8gb ram per sata connection), and 2.5/10gbe for smaller ceph clusters

You could buy drives with higher capacity, then you need fewer RPI, fewer cables, consume less space/energy and still achieve 55 TiB capacity.

Ceph recommends 1gb ram per tb of storage. Going lower gives massive hits to performance and the 1gbe ethernet connection becomes a bottleneck with fewer nodes, so I don't think it's viable to scale this way

True, the Pi 4 with 8GB RAM would allow for 8 TB max recommended storage per node. Thus, requires you to have 7 Pis for 56 TB. I think that's still a good setup, even with 1GE.

btw, 10GE is the minimum requirement for a non-hobby ceph cluster

Interesting, I don't know about Ceph, and now I must investigate this!

Before I built my current storage solution, I had been pondering something like this too, there's a certain allure to not have that "one box with all the drives", and be able to just expand forever by plugging more nodes on the network. I was thinking zfs over nbd, and ultimately decided against it, both because of the complexity, and becaure zfs wants to have ECC memory which the pi does not have (and for some reason I don't trust external enclosures to be as reliable as the PSU/Controller combo in a normal server).

Reply on news.ycombinator.com