Comparing Ceph, Linstor, Mayastor and Vitastor Storage Performance in Kubernetes | Hacker News Reader

Comparing Ceph, Linstor, Mayastor and Vitastor Storage Performance in Kubernetes

blog.flant.com

58 pointskvaps34 comments

Their ceph usage is quite odd IMO, that few disks per node and such a (for storage) slow network interconnect cannot give you good results. We have a two-year-old (time really does fly) ceph benchmark with 100G Ethernet interconnect for cluster network[0] if anybody is interested. IMO using NVMe's flash and then connecting that with anything below 25G makes not much sense and is a waste of money, as network isn't that expensive anymore; especially with three to five nodes you can set up a full mesh to avoid an often more expensive switch.

IME Ceph really starts to shine on a) slightly bigger setups and b) maintenance like upgrading nodes and swapping faulty disks, that's just a bliss if one doesn't violate a few basic rules (like maybe don't use 2/1 replica (two copies but return already after only one got written)). But also smaller cluster can work fine and can be quite performant, well at least if the interconnect bandwidth isn't one that was specified at the end of the 90s..

Also, ZFS 0.8.6 is from 2020, lots of stuff happened since then, a bit odd to post a new benchmark using almost two-year-old software.

[0]: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-be...

> Also, ZFS 0.8.6 is from 2020, lots of stuff happened since then, a bit odd to post a new benchmark using almost two-year-old software.

That's probably a side effect of using Ubuntu 20.04, just like having a 5.4 kernel (which is likewise getting old). Although yes, that probably isn't helping, and it would be interesting to do the same test and change nothing but moving to Ubuntu 22.04 and the corresponding kernel and ZFS versions, both of which should help.

It's absolutely a side effect of using Ubuntu -- I had to build ZFS (was quite easy) to get newer versions on Ubuntu and do a bit to prevent the system from picking up the default version (pakages & *-dkms packages)

"maybe don't use 2/1 replica (two copies but return already after only one got written)"

I don't think ceph ever does that.

Sure does, you can create a pool with that target/minimum replica just fine, there are some hosters that thought they were improving performance and data usage (which, well, is true), but ignored the impossibility of having any real redundancy at all; note that 2/2 is just fine though, but the pool will go read only if one failure domain goes down (compared to 3/2 where you can continue operating just fine in such a case).

IOW. 2/1 will never ever make sense, it's just crying for data loss, if you want to save on data usage, and if you also don't mind the (slightly) higher CPU load, you can use erasure coding instead:

https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ce... https://docs.ceph.com/en/latest/rados/operations/erasure-cod...

It still doesn't mean it accepts the write as finished when only 1 PG has it. I agree on the risks on why 2/1 is stupid, just not that this also means writes finish faster with min_size < replication_factor.

It does though, it defines when to complete the write command to a client and when general any IO should be allowed on an object (both need #copies >= min-size), from the Quincy PG settings docs:

  > Sets the minimum number of written replicas for objects in the pool in order
  > to acknowledge an I/O operation to the client. If minimum is not met, Ceph
  > will not acknowledge the I/O to the client, which may result in data loss.
  > This setting ensures a minimum number of replicas when operating in degraded
  > mode.

I have always wanted to make a ceph cluster with one disk per node, just to see how it would behave, from a performance view point it would suck, but from a fault tolerant viewpoint it would be amazing.

Note that they are using 10GbE network when even a single NVMe disk used in the test has few times more bandwidth so the bandwidth results are constrained by the network hardware.

Creating on-premise infrastructure a year ago we went for 2x25Gbps network. This or even 100Gbps seems to be current "mainstream" and 10GbE is definitely not enough for NVMe speeds available today.

3 servers with 2 disks each for a grand total of 6 disks?? This configuration is not even close to a suitable size for a tiny POC. How about 4 or 5 servers with a minimum of 8 disks each for a MINIMAL total of 32-40 disks? A more realistic size would be 8-10 servers with 12-16 disks each (total of 100-160 disks).

I thought all these object-storage systems were based on a Dynamo-like design, where you don't get the system's durability guarantees unless you've got at least 17 independent copies of everything.

To step back a level of consideration: if you're already running on an IaaS platform that has its own (expensive) object storage, can it ever be worth it to run your own object-storage system using that same IaaS provider's (expensive) compute? Or to run your object-storage system on some other bare-metal host, and then PUT things into it using your IaaS provider's (expensive) egress bandwidth?

That is the reason why bandwidth is expensive on cloud. So youll have no choice.

I wonder if the Linstor integration with Kubernetes got better. A few years back, the CSI driver was full bugs and you couldn't really use it from K8s alone, you had to use Linstor tools on the servers directly to get things unstuck or get orphaned volumes to actually get deleted.

A lot better.

A couple of years ago I could see volumes on Linstor getting completely stuck and unrecoverable whenever the network was getting busy or unstable. Nodes reboot were a nightmare too.

Have a setup now with their Piraeus operator[1], Kubernetes >= 1.20, rancher and calico, and it seems to be very stable. XFS have been giving better results too. Still, better not to try too many reboot loops on the nodes.

1: https://github.com/piraeusdatastore/piraeus-operator/

Note, kvaps actually wrote one of the LINSTOR operators (which is no longer maintained):

https://github.com/kvaps/kube-linstor

Yeah, because I have joined to Piraeus project :)

This is great. Does anyone have experience with building on premise scalable storage based on open source components ? I mostly read about problems and edge cases with Ceph in production. I would be grateful for sharing your resource, experience and recommendations on how to tackle such task.

I did have Problems with ceph many years ago but have many hyper-converged clusters that run ceph without any problems. In the last years I am running rook-managed ceph without any real problems.

When you use Ceph in conjunction with Kubernetes, most people probably use rook. It's supposedly been production grade for a few years, but I keep running into random corner cases. In the beginning it would have problems when you ran it with XFS(the default at the time) causing the kernel to lock up.

A lot of the corner cases nowadays are related to not being able to scale the cluster, odd behavior with CRD's of users and other components that require you to run ceph admin commands directly through a proxy. The annoying thing about a lot of the "Cloud Native" projects is that you will spend a lot of time going through github issue lists.

There is a Taiwanese company that provides extremely cheap OEM ceph racks though. I've noticed that in terms of maintenance and performance something like that or a cheap TrueNAS almost certainly ends up cheaper than production workloads, unless you don't care about the data integrity of your ceph workloads.

When you say open source components, do you mean more commodity consumer-grade hardware or just the software components?

What are the cases where NFS is not enough?

The usecases are related but difference -- these are block storage solutions, not filesystem level solutions. Some technologies work on multiple levels (Ceph), but others do not.

Any case where you need horizontal scaling or redundancy though data replication. So lots of cases.

Isn't NFS generally served by a single machine? I don't have anything inherently against NFS, but creating a SPoF / a box that can't rebooted doesn't sit well with me.

There are guides for HA NFS servers. I don’t have experience though. I assume that it requires shared block storage.

From personal experience, if you want to run some software that uses SQLite or similar...

ceph(and the others I assume) solves the same problem that raid solved, but using the network as a transport layer(rather than the computer for raid). this generally means it is very easy to scale.

So the answer to your question is that your question was wrong, you are comparing different layers.

you can actually access ceph over nfs, however usually you will try to use a ceph aware access method, ether the ceph filesystem driver, or the S3-like library depending on what your application wants.

I'm so glad we have cloud managed storage like EFS / EBS so we don't have to deal with this stuff...

I love to deal with stuff like that.

Sure, if you enjoy paying $80/TB-month (before replication or backups). If you're operating at a scale that justifies running a datacenter, EBS is basically a non-starter.

$80 is nothing compared to an infra engineer salary of +$5000 a month.

And you will need one. permamently. because this stuff breaks in so many spectacular ways... i.e splitbrain. a joy to resolve.

Per petabyte, we are talking $80,000 per month.

I love baremetal but I agree, distributed storage is hell. I found it slightly harder to sleep at night managing a prod Ceph cluster.

Reply on news.ycombinator.com