Hacker News Reader

I want to preface this - I don't have strong opinion already here, and I'm curious about Ceph. As someone who runs a 6 drive raidz2 at home (w/ ECC RAM) does your Ceph config give you similar data integrity guarantees to ZFS? If so, what are the key points of the config that enable that?

When Ceph migrated from Filestore to Bluestore, that enabled data scrubbing and checksumming for data (older versions before Bluestore were only verifying metadata).

Ceph (by default) does metadata scrubs every 24 hours, and data scrubs (deep-scrub) weekly (configurable, and you can manually scrub individual PGs at any time if that's your thing). I believe the default checksum used is "crc32c", and it's configurable, but I've not played with changing it. At work we get scrub errors on average maybe weekly now, at home I've not had a scrub error yet on this cluster in the past year (I did have a drive that failed and still needs to be replaced).

My RPi setup certainly does not have ECC RAM as far as I'm aware, but neither does my current ZFS setup (also a 6 drive RAIDZ2).

Nothing stopping you from running Ceph on boxes with ECC RAM, we certainly do that at my job.

Weekly scrub errors are definitely not normal. There has been a few bug fixes in Ceph & the Kernel. I would check how up to date your packages are.

Hardware errors are also possible but there has been a few software bugs so worth checking.

Here's a really curly one we recently found and solved at work (only in March) that causes both scrub errors and sometimes bluefs aborts, it's a kernel patch. Likely to happen under memory pressure:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

https://tracker.ceph.com/issues/22464#note-50

But there have been various others.

I've not had a single PG inconsistency on my RPi cluster in the one year it's been running. Weekly errors are at work.

Most of these are for actual disk media failures.

For 100,000 disks what is a normal failure rate in your opinion?

@mike_d you are right I made a very bad assumption about the size of the environment :)

@antongribok sounds like you have it in hand having seen your subsequent comments just didn’t want to leave someone assuming that error rate was normal in the smaller clusters that are more commonly but not universally the case. Evidently not the case for you :)

> Weekly scrub errors are definitely not normal

That depends on the scale of your cluster. The Backblaze drive report shows they lose 1.3% of their disks per year on average. Larger operations will have one or more people who just replace disks all day.

I was running glusterfs on an array of ODROID-HC2s ( https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/ ) and it was fun, but I've since migrated back to just a single big honking box (specifically a threadripper 1920x running unraid). Monitoring & maintaining an array of systems was its own IT job that kinda didn't seem worth dealing with.

Looking at that ODROID-HC2 i wonder when the drive manufacturers would just integrate such a general computer board onto the drive itself.

If you take say old i7 4770k's, how many of those along with how many disks would you need to get 1GB/s sustained sequential access with Ceph?

My single ZFS box does that with ease, 3x mirrored vdevs = 6 disks total, but I'm curious as the flexibility of Ceph sounds tempting.

I just setup a test cluster at work to test this for you:

4 nodes, each node with 2x SAS SSDs, dual 25Gb NICs (one for front-end, one for back-end replication). The test pool is 3x replicated with Snappy compression enabled.

On a separate client (also with 25Gb) I mounded an RBD image with krbd and ran FIO:

  fio --filename=/dev/rbd1 --direct=1 --sync=1 --rw=write --bs=4096K --numjobs=1 --iodepth=16 --ramp_time=5 --runtime=60 --ioengine=libaio --time_based --group_reporting --name=krbd-test --eta-newline=5s

I get a consistent 1.4 GiB/s:

  write: IOPS=357, BW=1431MiB/s (1501MB/s)(83.9GiB/60036msec)

Thank you very much for the test. That gives me a good idea of what I could expect from my hardware, which is not quite that beefy.

Is 9 the minimum number of nodes you need for a reasonable ceph setup or is that just what you arrived at for your use case?

I've seen setups with as few as 2 nodes with osds and a monitor (so 3 in total), but I believe 4-5 nodes is the minimum recommendation.

I would say the minimum is whatever your biggest replication or erasure coding config is, plus 1. So, with just replicated setups, that's 4 nodes, and with EC 4+2, that's 7 nodes. With EC 8+3, which is pretty common for object storage workloads, that's 12 nodes.

Note, a "node" or a failure domain, can be configured as a disk, an actual node (default), a TOR switch, a rack, a row, or even a datacenter. Ceph will spread the replicas across those failure domains for you.

At work, our bigger clusters can withstand a rack going down. Also, the more nodes you have, the less of an impact it is on the cluster when a node goes down, and the faster the recovery.

I started with 3 RPis then quickly expanded to 6, and the only reason I have 9 nodes now is because that's all I could find.

Can I ask an off topic/in-no way RPi related question?

For larger ceph clusters, how many disks/SSD/nvme are usually attached to a single node?

We are in the middle of transitioning from a handful of big (3x60 disk, 1.5PB total) JBOD Gluster/ZFS arrays and I’m trying to figure out how to migrate to a ceph cluster of equivalent size. It’s hard to figure out exactly what the right size/configuration should be. And I’ve been using ZFS for so long (10+ years) that thinking of not having healing zpools is a bit scary.

For production, we have two basic builds, one for block storage, which is all-flash, and one for object storage which is spinning disks plus small NVMe for metadata/Bluestore DB/WAL.

The best way to run Ceph is to build as small a server as you can get away with economically and scale that horizontally to 10s or 100s of servers, instead of trying to build a few very large vertical boxes. I have run Ceph on some 4U 72-drive SuperMicro boxes, but it was not fun trying to manage hundreds of thousands of threads on a single Linux server (not to mention NUMA issues with multiple sockets). An ideal server would be one node to one disk, but that's usually not very economical.

If you don't have access to custom ODM-type gear or open-19 and other such exotics, what's been working for me have been regular single socket 1U servers, both for block and for object.

For block, this is a very normal 1U box with 10x SFF SAS or NVMe drives, single CPU, a dual 25Gb NIC.

For spinning disk, again a 1U box, but with a deeper chassis you can fit 12x LFF and still have room for a PCI-based NVMe card, plus a dual 25Gb NIC. You can get these from SuperMicro, Quanta, HP.

Your 3x60 disk setup sounds like it might fit in 12U (assuming 3x 4U servers). With our 1U servers I believe that can be done with 15x 1U servers (1.5 PiB usable would need roughly 180x 16TB disks with EC 8+3, you'll need more with 3x replication).

Of course, if you're trying to find absolute minimum requirements that you can get away with, we'd have to know a lot more details about your workload and existing environment.

EDITING to add:

Our current production disk sizes are either 7.68 or 15.36 TB for SAS/NVMe SSDs at 1 DWPD or less, and 8 TB for spinning disk. I want to move to 16 TB drives, but haven't done so for various tech and non-tech reasons.

The async messenger has gotten the thread counts way down. May be a little easier now.

For the standard 3x replicated setup, 3 nodes is the minimum for any kind of practical redundancy but you really want 4 so that after failure of 1 node all the data can be recovered onto the other 3 and still have failure resiliency.

For erasure coded setups which is not really suited to block storage but mainly object storage via radosgw(s3) or cephfs you need minimum k+m and realistically k+m+1. That would translate to 6 minimum but realistically 7 nodes for k=4,m=2. That’s 4 data chunks and 2 redundant chunks which means you use 1.5x the storage of the raw data (half that of a replicated setup). You can do k=2,m=1 also. So 4 nodes into that case.

I would love to hear more about your Ceph setup. Specifically how you are connecting your drives and how many drives per node? I imagine with the Pis limited USB bus bandwidth, your cluster performs as more of an archive data store compared to realtime read/write like the backing block storage of VMs. I have been wanting to build a Ceph test cluster and it sounds like this type of setup might do the trick.

Each node is completely separate, housed in a good quality aluminum enclosure with a fan, and sitting on top of an external USB Seagate 2.5" portable drive (either 4TB or 5TB), connected via USB 3 cable. I'm pretty sure these drives are SMR, but they've been good to me, and they're fast enough for my needs.

Power is provided either using official RPi power supplies, or a couple of multi-port Anker USB power supplies that I had previously. A limit of 2.5 amps does not seem to cause any issues.

Currently everything is connected to a single switch, but I move things around my office sometimes, and sometimes have the RPis connected to two different switches.

Right now, everything including the 1 switch is connected to a single APC UPS, and that thing is super old, so that's another SPOF.

My clients currently are a few wired desktops and laptops over wifi, all connecting via CephFS. I haven't tested with librbd or krbd, I imagine it wouldn't be fast.

The RPis are mostly 8GB, but I do have a couple 4GB, and one RPi 400, which is kind of hilarious.

Everything is running Ubuntu 20.04, Ceph Pacific, and deployed from the first node with cephadm.

I use only Samsung microSD cards, either 32GB or 64GB. I don't think it matters what kind, but getting bigger cards makes me feel like they'll last longer. Most of the nodes have the var partition on the external drives (on a small partition at the beginning of the drive), but I do have a few where I didn't set that up early on, and haven't gotten around to redoing it.

I partition the drives and put LVM on manually, and tell cephadm to use the specific LV instead of the bare drive.

If you want any kind of performance, definitely set your expectations very low, but for me this works. I can stream at least a pair of 4K movies off this simultaneously, and I also run an instance of Paperless-NG off this over a CephFS mount and haven't had any issues.

I tried using Ceph twice at home. Once was via Proxmox, and it installed and ran perfectly fine, although tbf I didn't load it with much.

The next was via Rook, since I have a Kubernetes cluster, and it was a nightmare. I spent a week or so reading through all the docs I could find before I felt prepared to go through with it, only to have random clock sync issues that Reddit informed me were due to me enabling power savings mode in the BIOS for my nodes.

ZFS's biggest hiccup for me is when I do a kernel update and DKMS borks the upgrade. Other than that, it's been rock-solid. I run a normal and backup node with it, no regrets.

I solved the ZFS DKMS bork issue by moving to Debian 11 from Centos 8. I've had zero openZFS issues since the move. On Centos it would require work every time a sufficient kernel upgrade came in.

Since I'm familiar with RHEL I just swapped some of the Debian default services for RHELish alternatives (Firewalld, Podman, etc.).

I'm also using Debian; have been since Wheezy. I also template my VMs with Packer/Ansible though, so in the event it gets too messy or annoying to fix, I just export the pool and import it to a new VM.

The last issue I had was that linux-headers didn't get installed with the kernel update, so DKMS couldn't build on them. I assumed they were a dependency for zfs-dkms, but no.

> Plus with Ceph I can add just one disk at a time to the cluster and disks don't have to be the same size.

I'd like to note that ZFS now has RAID-Z expansion which allows us to do exactly that! It's an essential feature for home users since it allows us to gradually expand capacity instead of buying up all the storage up front at great cost.

I too researched ceph for this exact reason but was told the hardware requirements were too high for a typical home lab, yet you're running ceph on raspberry pis... I should probably look into ceph once more.

will come but not yet https://github.com/openzfs/zfs/pull/12225

I'm also running ceph (using the rook kubernetes operator) in my homelab. Been running this setup for 9 months now with 2 cheap HP elitedesk workstations i picked up on ebay and 2 8TB HDDs in each.

Since this setup has run incredible smooth so far, I plan on using SolidRun's HoneyComb LX2 as a ceph node with bigger disks and nvme write cache in the future. I looked at the raspberry pi 4, but was not too impressed by the single PCIe 3.0 lane, since I also plan on using NVME disks as ceph's metadata storage device to speed up the hard disk with the normal data behind it and the ceph recommendation to use 10GbE NICs.

The HoneyComb LX2 has 4 built in 10GbE ports, 16 A72 cores, actual DDR4 RAM slots, a 4 lane PCIe 3.0 m.2 slot and an open-ended (so you can put in a full x16 device) PCIe 3.0 slot with 8 lanes for a max of 8Gbyte/s bandwidth.

Since it's an arm box it's incredible energy efficient which is important since energy prices are increasing in my country. Also its the only affordable performant arm device at 800USD.

Hetzner used to offer vms backed by ceph block devices on hetzner.cloud, but got rid of them recently. Wonder why?