Ethernet SSDs – Hands-On with the Kioxia EM6 NVMeoF
servethehome.com
servethehome.com
Performance could have been a bit better, but this was back when 128gb enterprise SSDs were nearly the largest you could buy.
Right now a drive generally belongs to one computer. But disaggregating the rack & making disks independent means tbis association, of a drive or partition to a particular computer, is arbitrary. Potentially anyone could read or write to any drive.
This would require a clustered filesystem, a fs designed for multiple computers interacting with it at once. I believe there are some offerings out there, but mostly proprietary, and I've heard so little in the past decade- kt feels.like it should be cytting edge & exciting, not dormamt & specialized. I like to think we could get up to a lot of interesting computing & data systems if we had more io over fabrics.
A much more modest, pre-clustered idea might just be to heavily partition a drive, make 64 partitions, then build some control-plane of who currently gets which partitions. Letting chunks of data change ownership, without moving.
It'll be interesting to see if other fabrics than ethernet eventually start being useful with multi-root, multi-computer. PCIe is building new Scalable Virtualization, but as far as I know is still focused on being a peripheral bus, not interconnect. Most of the CXL discussion i see is similarly conventional- huge plans for build out, attaching vast memory pools and huge accelerators, but I havent seen multiple computers sharing a pool of resources shown off. I havent seen CXL as a way to interconnect standalone computers.
Unfortunately they went with 32bit PowerPC and it is no longer supported by Debian so no more upgrades unless I bust out a soldering iron and mess with u-boot to get something like netbsd on it.
Seems like a good product for the super cheap arm SoC makers, a board that attaches to the back of a hard drive with Ethernet just like WD did but without the big clunky case.
They supported a few key/value stores and claims a big win on less complexity and cool things like migration/rebalancing that could be done disk to disk. I think there was a dev kit for developing for the inside the disk platform.
Seemed like a cool idea, not sure what happened to it, I think it was for sale for interested parties, at least for awhile.
This isn't meant as a dig, but I'm genuinely curious as to what the use case for these things is.
As for the place it's really not much different from where existing large scale storage systems fit, just a different way of delivering them using newer transports which give a bit more flexibility.
>23 drives we can then RAID 0
umm, is this satire?
So we replace "expensive and complicated" storage controller with expensive consumables (ssds have limited lifespan) and move complicated part to the server. What are the upsides?
Basically the difference between 25GB/s between your server and it's storage and 25GB/s between your server and it's clients is just where the packets are going.
Pretty sure that's per drive.
This chassis has 1.2Tbps switching, half for drives, half for attached computers.
It's not proprietary. The drives can plug into ethernet. They speak industry standard NVMeoF; you could not get more standard if you tried.
So 24x 25GbE in that one switch chassis. The network-switch-chassis has 6x 100GbE (or is it actually 200GbE) for external connections.
Since the drives are dual-port, the other chassis has another 6x100/200GbE if you wanted to split traffic over the second disk port + network-switch-chassis.
Upsides seem to be:
- Can scale up storage by plugging in another switch/disk shelf
- No storage-enforced limits on the number or type of arrays
- Can define namespaces on drives so you can adjust storage to clients based on need.
- Can scale up performance for a single client by making more drives available and/or faster client NICs
- Not limited to the number of PCI lanes in the client so you could have fast access to FAR more disks than you could physically plug into one host. Performance would be limited to the client's NIC(s).
At the end of the video Patrick talks about that while they demo it being done directly on a host, they can also have DPU cards that could do all of the mounting/array management bits. There's a video on that coming soon.
Whats the appeal of proprietary drives talking to fiber switch directly versus NVME chassis with 3 2x100G PCIe NICs on the back? From logical point of view there is no difference. You can parturition, provision, bond it up just the same. You just move PCIE-fiber conversion from central point (cheaper) to bespoke solution inside proprietary drives (sounds expensive vendor lock-in).
I think Marvell just happens to be the first out with the NVMEoF drive adapter and switch chips.
Kioxia and the switch vendor seem to be first out with the drives and switches using Marvell chips/boards.
I wouldn't be surprised to hear in a year or two that you can buy a NVMEoF switch from a bunch of vendors, and then NVMEoF drives from your choice of drive vendor.
The specifics of advantages over existing solutions I can't speak to except repeating much of what is in the article and video.
I might be placing too much faith in Patrick + STH, but they seem excited, and other times there have been proprietary solutions for things they have pointed it out.
This solution proposes dropping a lot of hardware & getting the same outcome. There's one box with 12x 100Gbe on the back (across two switches) and a bunch of nvme (over rdma ethernet, "NVMeoF") plugged directly into that switch.
Why bother having a whole cpu & expensive nic just to connect some drives to the network? This box replaces not just that nas, but perhaps the top of switch box too.
This isnt supposed to be bespoke or proprietary. Its supposed to just the infra we already have. If your nas is really good it already spoke NVMeoF anyhow. Whats a little fancy is that you still need some PHY between switch and drive, which i think is throwing you off & confusing you a bit & making you think this is more bespoke than it is. The drive is an ethernet drive, could hardly be less bespoke.