Kioxia Demonstrates Raid Offload Scheme for NVMe Drives
anandtech.com
anandtech.com
[ 478.047970] raid6: avx2x2 gen() 60473 MB/s
[ 478.115971] raid6: avx512x1 gen() 53469 MB/s
[ 478.149971] raid6: avx512x2 gen() 57067 MB/s
Just seems like hardware RAID with all the same problems, likely not as fast as software RAID, harder to manage, a unique set of tools per vendor, harder to have global spares, and doesn't work with filesystems that do their own redundancy like ZFS.
These NVME drives can talk directly to each other for raid which means a much larger total bandwidth is available, and potentially improved latency also.
RDMA means you might not be serving via CPU at all.
I also wonder, if you have 8 NVMe, write to a stripe to one, it does the RAID calc and sends each disk the share of the stripe. What happens if the master NVMe dies? It's not really a RAID if a single disk can kill the RAID.
PCIe P2P transfers can go direct through a downstream PCIe switch such as found on chipsets without having to bump back up through the CPU.
Not to mention I'd expect the parity calculations to be MUCH slower on the NVMe controllers.
> upwards of 90% reduction in system DRAM utilization
stripe = read_from_ram(*ptr) # usually between 128k and 256k
blobs[]=do_raid_calc(stripe) # blobs usually 25% to 33% larger than stripe
for i in drives
write(drive=i,blobs[i])
> upwards of 90% reduction in system DRAM utilization
That seems unbelievable, most ram isn't spend for anything I/O related let alone RAID releated. Now if it's 90% reduction in system DRAM utilization by RAID, sure. But that seems like a very small fraction of all ram.
Even if 10,000 stripes are in flight simultaneously to 100s of drives that's only 2.5GB or 1% of a servers ram (256GB or more seems common). Especially since 2/3rd of that would be in ram even with hardware RAID. Not like the buffer/page cache which might reach 50% of ram has the extra RAID in data in it.
Sure, if you use X3D chip with the current largest amount of L3 cache accessible to a single core of any option currently available you can dedicate all of it to 128 MB of the write buffer to your disk instead of letting it be offloaded. Valid option, just as cool. I have a non X3D 7950X so jealous though ;).
You've also got the case of needing to transmit up the read of the disk for modifications to sectors not cached by the system so the CPU can perform the parity calc of the whole sector and issue the appropriate writes. Particularly bad for non-sequential IO writes.
> if it's 90% reduction in system DRAM utilization by RAID
Yes, this - not the other. It's achieved by not writing things back to RAM again before they hit the flash pool.
Ah, sorry, lscpu shows: L3: 64 MiB (2 instances)
I originally thought that meant 64MB x 2, but it means 64MB total (32MB x 2). Still 64MB is 500 times larger than 128KB stripe and I/O normally happens on a wide variety of cores, and should only be required for stripe that are in flight. Server (normally with 5x or more cores than my 12 core desktop) and way more bandwidth (24 channels instead of my 2) will have much more cache and much more bandwidth.
> Yes, this - not the other. It's achieved by not writing things back to RAM again before they hit (comparatively slow to RAM) flash pool.
Why should the stripes be written to ram? The write should enter kernel space (write is a system call), then the software RAID driver does the calculation and then the write to the devices memory space. The PCIe connected NVMe controller is not cache coherent and can't safely read main memory, which might be cached.
I took a closer look at the original post, they seem to be considering the tiny write, which requires a read/modify/write. Said operation is pretty inefficient, and linux tries to avoid this with caching, but certainly is needed sometimes. I've not seen any analysis on what fraction of I/O to production RAID system is R/M/W instead of a normal read or write.
Even in the R/M/W case, a stripe is read by the software-RAID driver, the write is masked onto the strip, and a new checksum is calculated. Then the stripe is sent back to the I/O space for each involved NVMe controller. So a 4KB write (common minimum size) requires reading 128-256KB, doing the checksum, and writing it back to the device.
It does tip the scales more towards hardware RAID, but that's always been true for hardware RAID, which very often ends up slower than software RAID for previously discusses reasons.
- Receive the new data
- Read the multiple disks to get the current stripe(s) associated with it.
- Calculate the new parity
- Issue the multiple writes
- Wait for completion, clear that from RAM
Looking at a single write it doesn't seem so bad. You take something like ~128k in from the disks per stripe (which will arrive it ever so slightly different times and be held as that thread stalls before the calc), issue a bunch of writes, wait for that to clear while the result remains in memory (cache or RAM), then you're good to clear it out and that thread/coroutine task can process the next one. "Just" 3 GB/s is ~23,000/s of that - doing those multiple reads into RAM, parity writes into RAM (well, unless you can stick it all in massive L3 by keeping queue depths low), and caching until spat out on to the drive. On a normal non-parity setup you just have your data to be written sit and go to disk, no intermediate reads/writes.
This may not make sense on a home box but consider the approach more an alternative to solutions like https://www.graidtech.com/product/sr-1000/ which are single cards that can get a million RAIDed IOPS written at near 100 GB/s in a single PCIe slot alone with no additional load to the CPU. Just writing 100 GB/s takes a CPU core and most of the RAM bandwidth from a raw data creation/parsing perspective before talking about writing it to disk at all, it's a different problem than e.g. what the bandwidth looks like on a home NAS pool. This type of approach tries to do something similar without the extra device in-between the cards and the server.
Sometimes you also want to take the above approach and scale it out over many 100G/400G ethernet ports so your flash storage pools are reachable over network separate from compute nodes. Here the goal is to make that storage solution as dense, fast, and efficient as possible where you might want to load as much possible storage as you can on a single node until it saturates the bandwidth to the CPU. If you can do that without doubling back data to the CPU you can scale it that much better.
Really? You somehow can't create a ZFS file system on an hardware RAID block device? Seems like that means the hardware RAID isn't the otherwise transparent block device it's supposed to be for the OS and whatever file systems it cares to employ.
You're concerns about management, tools and spares are correct for many use cases. Some uses cases, like cloud operators that don't suffer the burdens of long term management at that level of detail (where entire racks and generations of hardware are cycled in/out as a working unit, with ample spares at hand, under contract) won't care about that. They'll care about the nice efficiency gain. When you operate like that you can accommodate sophisticated integration such as this for efficiency gains.
Sure you can do it, have two layers of checksums and a volume manager on top of a volume manager. But ZFS is designed to talk directly to block devices and try to detect and complain about the numerous failure modes. Like say a parity calc that goes awry because of a memory error.
For this and other reasons it's recommended that even with Hardware RAID it's recommended to configure it in JBOD mode.
I've also seen numerous cases where software RAID on top of hardware RAID running in JBOD mode is faster than just using hardware RAID.
> When you operate like that you can accommodate sophisticated integration such as this for efficiency gains.
Sure, if there are efficiency gains. If the strong bottleneck for writing to the controller is your limiting factor you might get a 33% increase in I/O. But for that to be true you need:
* The bottleneck not to be elsewhere
* The controller inside a NVMe device (often passively cooled) to be faster than the one on the CPU
* The bandwidth between the PCI controller or PCIe switch and the NVMe controller to not care about a 2x increase in needed bandwidth
I'm still hopeful that we see it happen some day.
On the other hand, Windows and Linux still cannot just upgrade the vast majority of firmwares on NVMe devices, least of all consumer ones, despite being completely and utterly standardized.
You have to wonder, if Samsung makes bullshit, and then this https://github.com/chrivers/samsung-firmware-magic becomes part of the ecosystem, why trust the vendors with anything else?
There seem to be a lot of ISOs for download at https://semiconductor.samsung.com/consumer-storage/support/t...
The ISO contains an EFI filesystem with grub etc and boots fine without needing Windows.
Also i think i remember fixing (upgrading?) the firmware on a crucial ssd like 5 or 6 years ago using some live linux system (downloaded off the crucial website i think?)
Not sure about windows, but linux is getting incredibly better at this.
https://bughunters.google.com/blog/6303226026131456/a-deep-d...
I wouldn't want random applications (or web pages) to be able to load eBPF modules in the same way they can send shaders to a GPU through a graphics driver.
What I am the most puzzled by is how parity (i.e. RAID5) is so bad in windows storage space. A modern CPU should be able to xor data at several gigabytes per second. And it seems that even by optimizing the block sizes, windows storage space parity caps at a couple hundred MB/s.
It also doesn't change anything for distributed storage.
Can't afford data recovery. It was a backup drive though, so I need to redo the backups.
Thinking about buying more smaller drivers maybe on a couple mini PCs connected to network.
The drive still shouldn't fail entirely from a power outage, and should at most suffer data loss, but at the end of the day it's designed to be cheap rather than reliable.
I needed it for an experiment where I had about 6TB of small files to process and wanted to have them on a single drive. It did the job and then I repurposed it for backup / dump drive for stuff I didn't want to delete, but also didn't now where else to put it.
The drive shows up in the system but with 0TB capacity, I recall once or twice it reported 8TB but I was unable to read anything.
I'll have a look one day maybe that was something simple like dead cap that I could replace (I have microscope, rework station).
But they still fail. Backups are great and all, but for hardware-failure nothing beats redundancy (while RAID1, RAID5, etc allow for faster reads - I don't know how-often NVMe SSDs saturate their PCIe links though...).
Granted, you don't need hardware RAID for that (and HostRAID is a joke, lol): we still want redundancy, but today you'd do it with ZFS or similar so you aren't locked-in to some HW RAID vendor, or suffer the ironic consequences of having non-redundant HW RAID controllers.
When it drops off file-systems writes to the LVs are blocked and reads can also fail but the system survives sufficiently to do a controlled power off/on that recovers it.
In some cases the LVs pair up a spinning disk with the NVME but due to how I've configured the LV the spinner is read-mostly and the NVME is write-mostly (RAID member syncing is delayed and in background). There isn't too much noticeable latency except for things like `git log -Sneedle` - and worth it for the resilience.
[0] first time it happened it was spiders that had taken up residence around the M2 header and CPU (nice and warm!) and causing dust trails allowing current leakage between contacts (yes, I did do microscopic examination because I could not identify any other cause) that a simple blast with the air-compressor resolved. Later incidents turn out to be physical stress due to extreme thermal expansion and contraction as best as I can tell - ambient air temperature can fluctuate from 14C to 40C and back over 18 hours. Re-seating the M2 adapter fixes it for a a few months before it starts again! All NVME SMART self-tests pass; the failure is of the link not the storage - effectively being removed from the PCIe port. Firmware was at one stage suspected, although it had been fine for a couple of years on the same version, but updates haven't changed it in any way. ASPM is disabled.
We've had good experience with Xinnor, but it's a shame it's proprietary.
I'd love to see a high-performance open-source erasure coding solution for NVMe. The built in offerings in Linux are not cutting it.