Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
And remember that there's a multiplication of iops for any individual client iop, whether you're using triplicate storare or erasure coding. S3 also has iop multiplication, which they solve with tons of HDDs.
For big object storage that's mostly streaming 4MB chunks, this is no big deal. If you have tons of small random reads and writes across many keys or a single big key, that's when you need to make sure your backing store can keep up.
However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.
Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes.
So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast metadata store for Garage, the "special" device/small records store for the ZFS, the ZIL/SLOG device and so on.
Currently it's a bit of a Frankenstein's monster: using XFS+OpenCAS as the backing storage for an old version of MinIO (containerized to run as 5 instances), I'm looking to replace it with a simpler design and hopefully get a better performance.
TrueNAS can handle the OpenZFS (zRAID, Caches and Logs) part and you can deploy Garage or any other S3 gateway on top of it.
It can be an interesting experiment, and 80 disk server is not too big for a TrueNAS installation.
The link is a 10G 9K MTU connection, the server is only accessed via that local link.
Essentially, the drives being HDD are the only real bottleneck (besides the obvious single-node scenario).
At the moment, all writes are buffered into the NVMes via OpenCAS write-through cache, so the writes are very snappy and are pretty much ingested at the rate I can throw data at it. But the read/delete operations require at least a metadata read, and due to the very high number of small (most even empty) objects they take a lot more time than I would like.
I'm willing to sacrifice the write-through cache benefits (the write performance is actually an overkill for my use case), in order to make it a little more balanced for better List/Read/DeleteObject operations performance.
On paper, most "real" writes will be sequential data, so writing that directly to the HDDs should be fine, while metadata write operations will be handled exclusively by the flash storage, thus also taking care of the empty/small objects problem.
For software, at least with MinIO it's possible to do rolling updates/restarts since the 5 instances in docker-compose are enough for proper write quorum even with any single instance down.
? on the low end a single HD can deliver 100MB/s, 80 can deliver 8,000MB/s, a single nvme can do 700MB/s and you have 4, 2,800MB/s - a 10Gb link can only do 1000MB/s, so isn't your bottle neck Network and then probably CPU?
Same applies to your NVMe throughput since now you have the risk to congest the PCIe lanes if you're increasing line count with PCIe switches.
If there are gateway services or other software bound processes like zRAID, your processor will saturate way before your NIC, adding more jitter and inconsistency to your performance.
NIC is an independent republic on the motherboard. They can accelerate almost anything related to stack, esp. server grade cards. If you can pump the data to the NIC, you can be sure that it can be pushed at line speed.
However, running a NIC at line speed with data read from elsewhere on the system is not always that easy.
It would still need a resilience/cache layer like ZFS, though.
You're basically seek limited, and a read on a mirror is one seek, whereas a read on a RAIDz is one seek per device in the stripe. (Although if most of your objects are under the chunk size, you end up with more of mirroring than striping)
You lose on capacity though.
I think I'm going to go with 8x RAIDz2 VDEVs 10x HDDs each, so that the 20 drives in the internal drive enclosure could be 2 separate VDEVs and not mix with the 60 in the external enclosure.
Getting Ceph erasure coding set up properly on a big hard disk pool is a pain - you can tell that EC was shoehorned into a system that was totally designed around triple replication.
Now the first OSD splits it into k data chunks plus d parity chunks, so the disk write size isn’t 4MB, it’s 4MB/k, while the efficient write size has gone up 2x? 4x? since the original 4MB decision as drive transfer rates increase.
You can change this, but still the tuning is based on the size of the block to be coded, not the size of the chunks to be written to disk. (and you might have multiple pools with much different values of k)
There is also striping [2], which is the equivalent of RAID-10 functionality to split a large file into independent segments that can be written in parallel. Perhaps you are referring to RGW's default stripe size of 4 MB [3]?
If yes, I can understand your point about one 4 MB RADOS object being erasure-coded to e.g. 6 = 4+2 "parity chunks", making it < 1 MB writes that are not efficient on HDDs.
But would you not simply raise `rgw_obj_stripe_size` to address that, according to the k you choose? E.g. 24 MB? You mention it can be changed, but I don't understand the "but still the tuning is based on the size of the block to be coded" part, (why) is that a problem?
Also, how else would you do it when designing EC writes?
Thanks!
[1]: https://docs.ceph.com/en/squid/rados/configuration/bluestore...
[2]: https://docs.ceph.com/en/squid/architecture/#data-striping
[3]: https://docs.ceph.com/en/squid/radosgw/config-ref/#confval-r...
It's the classic horizontal/vertical scaling trade off, that's why flash tends to be more space/cost efficient for speedy access.
It's open source / free to boot. I have no direct experience with it myself however.
For single client performance, ceph beat the performance I get from S3 today for large file copies. Gluster had difficult to characterize performance, but our setup with big fast RAID arrays seems to still outperform what I see of AWS's luster as a service today for our use case of long sequential reads and writes.
We would occasionally try cephFS, the POSIX shared network filesystem, but it couldn't match our gluster performance for our workload. But also, we built the ceph long term storage to maximize TB/$, so it was at a disadvantage compared to our gluster install. Still, I never heard of cephFS being used anywhere despite it being the original goal in the papers back at UCSC. Keep an eye on CERN for news about one of the bigger ceph installs with public info.
I love both of the systems, and see ceph used everywhere today, but am surprised and happy to see that gluster is still around.
I used to keep a large cluster array with Gluster+ZFS (1.5PB), and I can’t say I was ever really that impressed with the performance. That said — I really didn’t have enough horizontal scaling to make it worthwhile from a performance aspect. For us, it was mainly used to make a union file system.
But, I can’t say I’d recommend it for anything new.
What performance issues and footguns do you have in mind?
I also like that CephFS has a performance benefits that doesn't seem to exist anywhere else: Automatic transparent Linux buffer caching, so that writes are extremely fast and local until you fsync() or other clients want to read, and repeat-reads or read-after-write are served from local RAM.
What you mean by no EC?
Seems like some kind of compression?
Is that how the error correction on DVD works? I
And is that how GridFS is can keep file store slow low compare to regular file system?