Hacker News Reader

Comment by 4by4by4 | Hacker News Reader

We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data.

Seaweed had much better performance for our use case.

Do you wish it supported Erasure Coding for lower disk usage, or is your workload such that the extra spindles from replication are useful?

That would be nice and that’s why we first tried MinIO.

But with MinIO and erasure coding a single PUT results in more IOPS and we saw lower performance.

Also, expanding MinIO must be done in increments of your original buildout which is annoying. So if you start with 4 servers and 500TB, they recommend you expand by adding another 4 servers with 500TB at least.

Forgive my ignorance but why is this preferable to a big ZFS pool?

Not the only reason, but we have a distributed workload so HTTP is a better protocol than NFS for our use case.

Internally ZFS is more or less an object store. I seem to recall there was a project to expose this to users directly, similar to how ZPL (filesystem) or ZVOLs are just two different ways to expose the internal object store.

Seems like it could be an interesting approach.

I could be wrong here, but I believe this (ceph, et al) is the answer to the question: > """But what if I don't have a JBOD of 6x18TB hard drives with good amount of ECC RAM for ZFS? What if I have 3 raspberry pi 4's, at different houses with 3x 12TB externals on them, and 2 other computers with 2x 4TB externals on them, and I want to use that all together with some redundancy/error checking?" That would give (3x3x12)+(2x2x4)=124 TB of storage, vs 108TB in the ZFS single box case (of raw storage).

If you could figure out the distributed part (and inconsistency in disk size and such), then this is a very nice system to have.

"What if I have 3 raspberry pi 4's [...] with no significant performance requirement?"

Ceph is nice, but performance is lackluster on anything but a proper cluster (pun intended).

It's also somewhat heavyweight. I ran ZFS over iSCSI with four RPis serving the iSCSI targets via SATA-USB. It was network limited mostly. The advantage of that is that you can take the same disks and plug them all into a single host and import the pool directly (ie not via iSCSI), if needed.

I would love to learn more about that setup. Do you have it documented anywhere?

Unfortunately I have not.

The core concept is rather simple though: iSCSI allows for an iSCSI server to expose raw block devices (called targets[1]), so do that and use them from a client machine as-if they were locally connected disks.

So I used LIO[2] as the iSCSI server on each of the Pi's, making sure to use the "by-id" to reference the block device so it would work fine across reboots (skipped changing "directories"):

    sudo targetcli
    /backstores/block> create name=block0 dev=/dev/disk/by-id/usb-SAMSUNG_MZ7PD256HAFV-000_0123456789000000005-0:0
    /iscsi> create
    /iscsi> cd iqn.2003-01.org.linux-iscsi.rpi4.armv7l:sn.fc1b1c9879a1/
    /iscsi/iqn.20....fc1b1c9879a1> cd tpg1/luns
    /iscsi/iqn.20...9a1/tpg1/luns> create /backstores/iblock/block0

Note you might have to use /block/ instead of /iblock/ in the last command there[3], depending on targetcli version.

Then you create the portal which exposes the target (backed by the raw disk) to the network as normal (see Debian guide fex). I did it like this, you have to adjust for the name of your device:

    /iscsi/iqn.20...1c9879a1/tpg1> cd portals
    /iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 3260
    /iscsi/iqn.20.../tpg1/portals> create 10.1.1.101
    /iscsi/iqn.20.../tpg1/portals> cd ..
    /iscsi/iqn.20...1c9879a1/tpg1> set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

This creates a portal which exposes a single LUN, backed by the raw disk, on the IP 10.1.1.101, and disables authentication and demo mode (write protection).

Next you install the iSCSI client (called initiator)[4], I did this on a separate machine. There I connected to each of the target on each Pi. Once successful you should get a number of disks under /dev/disk/by-path/ip-*, which you can then specify when importing the pool on the iSCSI client machine, say

    zpool -d /dev/disk/by-path/ip-X -d /dev/disk/by-path/ip-Y

Since you're exposing the raw block device, there's no difference in that regard to having the disk plugged into the client machine directly.

Btw, I recommend not messing with authentication at first, though it's not terribly difficult to set up.

edit: As mentioned the nice thing about this approach is that it's effectively "harmless". If it doesn't work out for you, you can always put all the disks in a single machine and import the pool as normal.

Also, iSCSI has a lot of robustness built in. For example the iSCSI client will temporarily store writes and re-issue once the target (server) is back online. I safely rebooted one of the Pis while copying data to the pool, for example.

[1]: https://en.wikipedia.org/wiki/ISCSI#Concepts

[2]: https://wiki.debian.org/SAN/iSCSI/LIO

[3]: https://github.com/ClusterLabs/resource-agents/pull/1373

[4]: https://wiki.debian.org/SAN/iSCSI/open-iscsi

its distributed: will survive if your server dies..

Because you need an S3 compatible API?

I use ZFS for most of my things, but I have yet to find a good way of just sharing a ZFS dataset over S3.

Any hiccups?

Drop in S3 compatibility with much better performance would be insane

Setup is a little obscure, but the developer is responsive on Slack and GH.

We are only a couple months in and haven’t had to add to our cluster yet, storing about 250TB, so it’s still early for us. Promising so far and hardware has already paid for itself.

why not ceph?

Reply on news.ycombinator.com