A Ceph war story

>Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems by just removing the XFS dependency for the actual data. Recommended reading: https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf

This gave me a concern. My kube nodes do use XFS in some cases but Ceph uses raw block devices. So XFS is only used for system files, not for Ceph. Except of course to store Ceph config on each node.

So I assume I'm safe. I'm not entirely sure how you'd use XFS with Ceph because Ceph uses a raw device file and formats it for its own storage.

> Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode.

well raid 1 works aswell, if you can take the performance hit. most raid 1 drives inside a raid controller can be run outside of the raid. we already needed to do that since one of our customers tought it is a good idea to have a running server temporarly near an open window with production data and no backup (we exclude our liability in this case and monitor if backups are created) so we needed to recover the data. which worked by using another controller with passtrough and just running a single disk. (the other disk was destroyed, so was the raid controller) btw. rainwater damages a server, especially if you do not notice it for 30 minutes and a full bucket of water inside it. kudos to dell the server kept running for 25 minutes, when it was full of water until it died. (we transported the server and still had water in it...)

Interesting read and helpful lesson list.

I've only used Ceph as provided to be by others and considered setting it up in some instances. Didn't know about the development of BlusStore and it does seem much simpler. The choice between xfs, btrfs, ext4 always seemed a bit unclear (except that I had experienced non-Ceph troubles with btrfs).

Note to self: use ceph-volume/BlueStore.

ceph-volume still relies on LVM, which brings unnecessary complexity.

We'd like to stick to ceph-disk(already unavailable in the P release) with raw block device only.

> fancy RAID controller

I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster.

Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs.

The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to put it in the trash and get something that does. If Ceph doesn't really work, RAID hardware is just adding an extra failure point that adds nothing.

The fancy RAID controller just shouldn't be there - there's no reason to use it unless you're stuck with it. Sometimes you can flash these LSI controllers with HBA/simple-RAID firmware ("IT" or "IR") to fix this.

But really, this had little to do with Ceph (other than the ops mistake of rebooting several machines at once without waiting for an all clear, which is just a bad idea all around when you're upgrading a cluster of any kind). It was an interaction between XFS (not used on modern ceph-volume systems), the kernel, and the RAID controller.

I find Ceph to actually be a lot more introspectable than "typical" multi disk storage management filesystems like btrfs and zfs. On those, if something goes wrong, you're often left with a corrupted or unmountable filesystem, or worse, kernel panics or errors. It's all a monolith in the kernel and hard to fix anything. On Ceph, you can dig through the layers, and things are split into separate daemons that can be debugged separately.

For example, want to see where the data of a CephFS file is stored? Easy: the inode number is the first part of the object name in the data pool. Append the block number after that and you have your object name. You can just fetch that using the rados tool, bypassing CephFS entirely. Want to go deeper? Ask it to hash the object name to a PG ID. Use `ceph pg dump` to figure out what OSDs that pg lives in. Now you know what disks have your data. OSDs broken? Use the objectstore tool to read the data directly without booting an OSD.

Obviously you need to learn all this stuff, but the tooling is there and this is way more introspectable than a bunch of magic structures in a filesystem.

To be fair, it does say:

> Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability.

although it's annoyingly hidden away in a carousel. Carousels - not even once.

(Author here) Yes, it was an XFS/controller issue, but Ceph reported the failure. :) (IMO it wasn't really a good decision from Ceph to use 100MB XFS partitions as a kind of database, but nowadays ceph-disk (which uses those XFS partitions) is gone, and instead ceph-volume uses a different approach via LVM.)

Regarding configuration management/firmware version: yes - especially, as you'd need to also rebuild disks in the dev/test environment with the identical configuration (firmware, disks,...), to ensure it's actually identical. And even if we neglect load/capacity/usage issues (problems might show up only under specific work loads), there are also further "invisible" layers/components like cables, NICs, switches,… and their firmware versions which are also relevant. Not exactly trivial. :)

I (with 2-3 other people) run over 100 PB on Ceph clusters in production, supporting some critical functions of a Fortune 50 enterprise.

We use Ceph for block and object workloads (no CephFS). Most of our clusters are still on Luminous (v12) and Filestore (XFS), and only our newer clusters that are being built now are on Nautilus (v14) Bluestore. We plan on migrating to Bluestore this year (and likely next year).

I'm on-call basically all the time, but we'll have maybe 1 issue per year where we have to act immediately. Most failures that would happen on a Saturday can wait until Monday to be acted on.

The smallest cluster that we'll build is 6 nodes. Our largest cluster now is 120 nodes with 1920 OSDs. We might build a larger cluster this year.

Back in Ceph Hammer days, I had a 6-node cluster lose 3 nodes, one node at a time, over a course of a couple of days with zero downtime.

We deploy with Ansible, and have our own, very paranoid and opinionated playbook for doing a rolling cluster reboot or restart, that checks multiple things before moving on to the next node.

Ceph user here. Ceph works fine 99.9% of the time until it doesn't. In this case you get a bit panic and start to google, ask in IRC for help and take a look at the bug tracker. In all critical cases of the past we were able to recover and gained ceph experience. I guess this is how you become a ceph expert.

I run a home ceph cluster as a hobby. It works really well. I have hit a few snags, but the mailing list is super friendly and helpful and the devs are quick to respond to bug reports.

Reading other comments here, I guess I am lucky enough Ceph still works fine for me, however I am still baffled by its potential performance. Last cluster deployed is 3 nodes each with 4x P4610 NVMe as OSDs, 25Gbps network pushes only 20k IOPS during 4k read, queue depth 128. Single disk is supposed to push around 600k...

“One of the issues is that the officially recommended way to install it seems to be rook-ceph with kubernetes.“

I believe the recommended way is using cephadm.

I have been using ceph-ansible since luminous and had to deal with nasty issues or quirks as well caused probably by complexity and human errors.

XFS has a history of reclaim/memory management issues. It used to have a habit of blocking on I/O when cleaning memory even with plenty of clean page cache available.

For the longest time, I had issues on my home server (XFS on top of RAID at the time) with large latency hiccups, to the tune of several seconds. This was disrupting some real-time data ingestion and causing data loss. I thought it was about committing to disk, so I added big memory buffers, but no dice. I spent years with this annoying issue. I even had a kernel patch in to increase kernel-side buffers (which were not subject to this problem) to work around it. It wasn't even consistent.

Eventually one day I got sick enough of it, and sat down trying to reproduce it. I figured out that it only happened when true memory usage (including buffer cache) was ~100%; if there was truly unused memory around, things were fine, and I could evict the buffer cache and it would fix the problem until it grew to consume all free RAM again. Eventually I managed to get a stack trace of a process that was stalling even though it wasn't anywhere near writing to disk, and I found out it was stalling during a write(). To a pipe. Because the kernel had to allocate data for the pipe buffers. And it was asking XFS. And XFS decided to evict some dirty inodes, and block on that. Even with gigabytes of clean page cache available to evict. What.

Swearing ensued - here I thought I had some weird kernel/hardware issue causing latency spikes, and it was XFS all along. I eventually ripped XFS out and replaced it with ext4 and that solved the issue.

This eventually got fixed in 2019: https://lwn.net/Articles/795098/

Gluster is also so much easier to set up IMO; the only reason I wouldn't use it is that it seems to have some consistency issues such that ex. you could run postgres on ceph, but not (safely) on gluster. But I don't understand it well enough to know why.

I was a Gluster developer for about ten years, including time as a project maintainer and as part of the team responsible for the largest deployment in the world. I also worked professionally with Lustre for a couple of years, and have some familiarity with Ceph.

Ceph and Lustre are both largely "set and forget" for the object-storage nodes which are the most numerous type. On the other hand, both have separate metadata servers which can be much more troublesome if/when one fails. Gluster only has one kind of node. One of the main issues here is: when do you actually give up on a node and start regenerating its data somewhere else? Starting that process and then having the node actually come back can be anywhere from a waste of time to a total disaster, so you don't want to be too "twitchy" about it, but you also don't want to run in a degraded state forever. In Gluster at least, the approach was to assume that a node's coming back unless/until explicitly told otherwise (possibly by external automation).

The larger point IMO is that no distributed storage system is "set and forget" at any significant scale. At the very least you'll want robust monitoring (which to some extent has to be built into the code) and alerting, and somebody to respond to the more serious alerts. I'd give a nod to Ceph in this respect. Even better is to have your own site-specific automation for common tasks like capacity addition and upgrades. Even if the storage system itself is doing everything "right" it can get pretty messy if that's not happening in sync with other systems such as provisioning and service discovery - not to mention the systems actually using the storage.

Also, even though the "POSIX is dead" folks are my sworn enemies, it's still true that an ever-decreasing number of workloads actually require an actual filesystem and its associated complexity. Not zero, probably never zero, but smaller every day. If you reasonably can get away with deploying a simpler kind of storage, I recommend it.

Lustre is not aligned at all with your requirements, so forget that one.

Ceph is much more complex than Gluster, but also more capable.

Honestly unless you are dealing with hundreds of TB of storage (and therefore need multiple servers anyway), I expect the complexity any distributed file-system adds is going to be detrimental to uptime and stability more often than it provides extra resilience. Use a single box with ZFS if you can, and add Gluster on top only if it can't be avoided.

I have mildly bad experience with gluster. I did never lose data, but the maintenance load was very noticable! Ceph will rebalance by itself when adding (or removing) disks. That alone reduces maintenance by a huge amount. For Storage clusters I don't care about as much, it's also nice when a disk fails that ceph will notice and react automatically. Replacing the disk is then something you ought to to at one time, but you don't have to do pretty soon.

I am running some rook-managed ceph clusters now that make even ceph infrastructure nodes automatically recoverable. But I did not test that enough yet.

i only have experience with ceph and would argue that if you set up a cluster with version 15.x right now it is pretty close to set and forget.

No! Not at all. You have to control uptime of the single nodes holistically. That just does not work with Ceph. I can't imagine that there are storage systems that work with that setup.

(Author here) Because it was absolutely unclear at that time yet, that an older kernel version could mount the XFS partition but the newer kernel versions could not (this only came up during the post mortem/RCA later). Furthermore the clock skew and mon_host issues gave us a wrong picture of the situation. (Also as the hosts are running as hypervisor systems, the kernel version should ~match with the environment (Proxmox/KVM), so there might be other/unclear risks with running such a setup.)