Hacker News Reader

Comment by throwaway2037 | Hacker News Reader

throwaway2037

I'm confused by this part:

    The system can run on any HDDs, but primarily runs on Shingled Magnetic Recording disks.

SMR is just one of the latest HDD technologies. Wiki tells me that is is about 10 years old now. I cannot believe the implementation of the storage hardware is going to have any affect upon how the software service runs. What am I missing? This sounds like a bit of "tech mumbo jumbo" on the lead-in. To be clear: I am not doubting the impressiveness of this system!

The conclusion is also pithy. I like it.

    Managing Magic Pocket, four key lessons have helped us maintain the system:

        Protect and verify
        Okay to move slow at scale
        Keep things simple
        Prepare for the worst

View on HN

Woah, I just found this on the SMR Wiki page: https://en.wikipedia.org/wiki/Shingled_magnetic_recording

    The higher density of SMR drives, combined with its random-read nature, fills a niche between the sequential-access tape storage and the random-access conventional hard drive storage. They are suited to storing data that are unlikely to be modified, but need to be read from any point efficiently. One example of the use case is Dropbox's Magic Storage system, which runs the on-disk extents in an append-only way. Device-managed SMR disks have also been marketed as "Archive HDDs" due to this property.

Cool! I was probably wrong in my previous post: The hardware implementation may truly affect the software.

It's not mumbo jumbo at all. Please read the rest of that wiki article. This is not a regular HDD. Buy a few, build a RAID out of them, then try to rebuild a drive while the front-end is online. You'll probably get data corruption and everything will crash.

It's an HDD technology that has to be combined with very specific type of logical volume management and protection workloads, and is very cheap at the tradeoff of these limitations.

To build a large, fault tolerant storage system on top of this "just another HDD technology" as you call it, is not something you can do at home with your normal tools. They did it, and they saved lots of cost because of that.

Do you ever walk into a meeting with an engineer SME talking about a topic which you had to look up on wiki, and because you don't understand what he's saying you dismiss it as "tech mumbo jumbo?" You will be very successful as middle-management at a large corporation.

> and is very cheap at the tradeoff of these limitations

Where can I get SMR drives that are a good deal? And what percent price improvement are you seeing?

When I look at the drives I can easily buy, the bigger models are all CMR and the smaller models keep having SMR snuck in without any notable price reduction.

I suspect they only bother selling large capacity SMR drives to hyperscalers, because at high capacities (eg. 8TB+), the workload for non-hyperscaler customers is usually something like RAID5 or ZFS, which is terrible on SMR drives. Hyperscalers don't have this problem, because they have custom software that can use features like ZNS to make SMR drives bearable, and on the lower end, there are enough "normies" (for lack of a better term) who are just using their drives to copy files back and forth, and probably won't hammer the drives with RAID5 rebuilds or ZFS resilver.

Are RAID5 rebuilds a problem? I thought the issue with ZFS was how scattered the writes are (with work happening on improving that), and RAID should just be a linear sweep.

No idea where you or I can get them, but for low-tier enterprise archival storage, vendors either put them into their own archival-tier arrays, or buy directly from the HDD vendors.

I think of SMR like Optane. A bunch of consumer-level tech guys w/o a real job got all lit up and started arguing about it online, but its target market was not the "NewEgg website" market, it was the "million dollar storage array from IBM" market.

Remember, there are many workloads that are "write this on disk and keep it forever" - even historical data in a data warehouse showing soda sales by season.

I honestly haven't bought any storage or compute in 25 years - my personal crap is old "enterprise/business line" junk from work.

Where you, as a consumer, can get them - don't. I don't think they're meant to be used by consumers. You're gonna have a bad time unless you write software specific to this weird type of drive. What you might find out there is some kind of a storage 'consumer NAS in a box' product that has tiered storage inside, with these drives and other ones. But stay away from drives on their own w/o that special software in front.

Though as a consumer it was easy to buy an optane card.

Not the dimms as much but those had horrible prices per gig, and SMR is supposedly a way to save money.

> Where you, as a consumer, can get them - don't. I don't think they're meant to be used by consumers. You're gonna have a bad time unless you write software specific to this weird type of drive. What you might find out there is some kind of a storage 'consumer NAS in a box' product that has tiered storage inside, with these drives and other ones. But stay away from drives on their own w/o that special software in front.

That would be a reasonable statement if they weren't selling SMR drives to customers without even labeling them as such. Just raw drives that sometimes have the performance go to hell. If you wanted fast you should have gotten an SSD, I guess.

It's not that I can't easily get an SMR drive, it's that I can't get the bigger models and I can't get the price savings.

Author here. We recently published a post about our last 4 years on SMR usage here: https://dropbox.tech/infrastructure/four-years-of-smr-storag.... Note that SMR technology just 5 years ago was rather nascent and a lot of software support was not great. Using SMR for us is possible without penalty only because of our sequential write workloads.

We use a custom disk format along with libzbc, but libzbd now provides many advantages, which we are looking to adopt. I did want the QCon talk to have some super straight-to-the-point conclusions and these, I believe, have saved us the most since I have been on the team. Largely due to the sheer scale of managing such a system.

SHR has high latency when you need to rewrite data on a shared "shingle" - so my guess is they designed the storage system to avoid those for better performance?

Reply on news.ycombinator.com