Comment by techdragon | Hacker News Reader

Comment by techdragon | Hacker News Reader

Parent Full thread

This is why I’m looking for more info. This is talking about how they are already sending these out to Hyperscale customers, and they will eventually be in individual customers hands… so it would be good to know what file systems and cache setups are needed in order to benefit from this.

I used to just rely on FreeNAS but it’s not as straightforward anymore. I’m having to consider Linux and looking at bCache FS and ZFS vs BTRFS and how all this compares on an all PCIe (m.2) flash drive setup… where a drive (or two) at this sort of size (~50TB) would make a great second backup copy of the flash array that can be started and stopped to periodically make the backups to save power… but then you have to think about copy efficiency since I don’t want to have them wasting read bandwidth from the flash drives… and the best is usually like to like file system copy ZFS -> ZFS and BTRFS -> BTRFS … so it adds another complication into a mix that is already far from simple.

So it’s become something I’m keeping an eye out for… hopefully before it becomes something I may purchase, someone will have already done a good writeup.

> … so it would be good to know what file systems and cache setups are needed in order to benefit from this.

I’ve worked on these systems. You might as well be asking F1 drivers for tips to help you commute to work.

The hyperscale stuff is not built on top of ordinary filesystems. It’s all clusters of machines, and error correction is handled at the level of clusters. If you’re evaluating systems like ZFS and BTRFS, then you’re already working with a radically different tech stack.

At scale, your file metadata is stored in a distributed database of some kind, and the file contents are stored with forward error correction across multiple machines in a cluster. Or something similar, but stored across multiple clusters. The pricing models for cloud storage are designed around the usage patterns—like, if you know that some data is going to stick around for 90 days, then SMR is a win. If a file could get deleted at any time, then SMR is a loss.

> You might as well be asking F1 drivers for tips to help you commute to work.

Boston drivers: "hold my beer."

The simple answer is that the software stack necessary for consumers to effectively use these storage devices does not yet exist. Hyperscalers are pretty much defined as the ones who are big enough to do their own software stack. For the rest of us, we have some good components to work with but also some major gaps that won't be filled anytime soon.

ZFS is a great improvement over traditional hardware RAID systems, but is still in many ways clearly a descendant of them. BTRFS has a slightly different mix of features from ZFS that make it a bit more flexible and a better choice for consumers who don't buy drives by the dozen. Neither has a great solution for caching/tiering with SSDs and hard drives. Ceph has a lot of features that would otherwise be almost exclusive to the hyperscalers, but is too complicated for something like a turnkey NAS. bcachefs aspires to eventually have most of the features you would want for a non-clustered storage system.

Zoned storage is something many of the above already have some degree of support for, but as a paradigm it has not even started showing up in the consumer computing ecosystem so many of the issues with adopting zoned storage for consumer systems aren't even being worked on.

ZFS has the concept of a separate intent log device though, I wonder if that would be enough to make these types of disks work with it? I have no problem putting a couple of terabytes of flash in front of a ZFS array if it means I can have 40TB of redundant storage in 3 disks.

The ZFS intent log is an example of a supremely disappointing, underwhelming way to integrate SSD storage into your system. Fundamentally, it's just a workaround for the fact that most of the write caches preexisting in the storage stack are volatile caches and thus not safe. Putting the ZIL on an SSD allows you to get the safety without sacrificing the performance benefits of the write caching (performance that everyone has come to rely on). But the ZIL doesn't help with read performance, and the data you write to it is never even read unless you have a crash or power failure. And then for the read caching, ZFS has L2ARC as an entirely separate feature. So ZFS does technically have ways to take advantage of SSDs to improve a hard drive storage array—but I don't think anyone would consider it to be an ideal solution, more of a pragmatic minimum viable feature.

Y'all keep saying "these" but SMR and HAMR are unrelated.

In theory, yes, but the bottom of the article explains that Seagate currently plans for their upcoming 24TB drive to be the last new non-SMR drive and all larger drives (28+ TB) are planned to be SMR. So in practice SMR and HAMR will be going hand in hand, at least from this vendor.

That's not my interpretation but we'll see.

> "So, we have a 24TB coming out soon, next few months, you will see it," said Romano. "That is the last PMR product. So I would say [higher] capacity point above 24TB PMR, that is probably 28TB SMR."

Sounds like the only uncertainty is in what the next capacity past 24TB will be, but it'll definitely be SMR rather than PMR.

I agree this part is a bit ambigous, but it says somewhere before this that the next drives to be released will be a 22TB and 28TB SMR drive. So I read this comment as "after this last 24TB PMR drive there will be the 28TB SMR drive before we will see the 32TB+ HAMR drives going forward" - so to me he was more talking purely about upcoming sizes rather than technology connections and SMR seems to be on its way out too!?

Reply on news.ycombinator.com