Hacker News Reader

Comment by johnchristopher | Hacker News Reader

That's a problem. I have been using and recommending WD to everyone who asks me what drive they should buy (based on backblaze data and 20 years of personal experience) for their private and everyday usage. Even though these are NAS drives it tarnishes the brand.

View on HN

I have stopped caring about hard drive brands.

What Backblaze and experience show is how drives of a certain brand performed a couple of years ago, not how the drive you just bought will perform.

IBM used to be very good until Deathstars, and now HGST, who took over is among the most reliable. I used to be very satisfied with Seagate, then ST3000DM001 happened, and now, it looks like they are slowly getting back on track. Maybe it is WD turn right now. I don't think there is a way to know until it is too late.

Personally, I like to make RAID mirrors with drives of different brands, or at least different models. Even it it is not ideal for performance, it helps making sure that not all drives fail at the same time.

>and now, it looks like they are slowly getting back on track.

The smartmontools ticket linked in the article says that Seagate is doing the same thing. Best to avoid.

Indeed. According to a newer article [0] on the same web site:

> Some Seagate Barracuda Compute and Desktop disk drives use shingled magnetic recording (SMR) technology which can exhibit slow data write speeds. But Seagate documentation does not spell this out.

---

[0]: https://blocksandfiles.com/2020/04/15/seagate-2-4-and-8tb-ba...

does WD still make new HGST products? How can you tell which drives are from which team, or have they been blended together?

Some HGST tech is Toshiba now, some of it is WD. Hard to know whether WD drive is made up to HGST snuff. If you want the best, buy helium WD drive labeled HGST. If you can't get it, then WD or Toshiba helium datacenter drives should be close.

That's what I'm wondering though. Nearly all the WD data center drives are marketed as Ultrastar. Is this marketing, or did the HGST entity supplant WD enterprise during "integration" in the last two years? https://www.westerndigital.com/products/data-center-drives#h...

It looks like the only surviving product names are Ultrastar and CinemaStar. Nothing is labeled HGST anymore. Most of the HGST Helium disks were SMR, if I'm not mistaken. As of now, given the two HelioSeal series, one of the two is SMR.

Side Note: Hitachi had such a great thing going with their Schoolhouse Rock meets Dino DNA commercial. I wish this would have continued. https://www.youtube.com/watch?v=xb_PyKuI7II

i do that too, and effectivele that means i had to get one of each brand available. the choices are really limited here.

Not so sure that helps in this situation. FreeNAS users are reporting these drives failing to resilver, which _could mean_ that you end up with a failed array and data loss instead of a failed drive. I'll personally be staying away from WD for a while. I'd rather drives fail prematurely than be a ticking time bomb.

The news isn't that WD is doing this, it's that WD is also doing this. At this point both the top 2 are doing this.

I'm so sick of HDDs, I can't wait for SSD NAS to close the gap. An NVMe M2-blade NAS would be incredible!

I don't trust ssds for long term storage, and I'm mortified on the experience when they die from one second to the other.

At work, we had literal hundreds of machines failing the same week because of sudden ssd death.

Ssds are fine for speed, but for storage... maybe one day.

Probably not. The permanent need for power to avoid data slowly disappearing from the device pretty much makes SSDs a non starter for long term storage.

Surely routine ZFS scrubs would mitigate this?

SSDs literally zero themselves out if they’re left unplugged for too long. No file system trickery will save you from that.

Are you sure? This article says that was a misunderstanding.

https://www.pcworld.com/article/2925173/debunked-your-ssd-wo...

That’s not what the article says. The article says that consumers are overreacting, but that SSDs can and do lose their data with loss of power and/or heat.

I feel like we are a long way off from that just based on price. Generously I think we can say a 1TB SSD (sata even to go cheaper) is about $100. That compared to a bare WD Red 8TB which comes in at ~$27/TB. ~4x the cost and smaller drives is a hard sell (even with the speed). I have a pool of SSD's that I keep some stuff on but the rest goes to spinning rust.

There are really only two or three HDD makers out there, so you really can't take brand too seriously.

IBM had one batch of bad drives and a second batch of somewhat iffy drives after many years with an excellent record and suddenly nobody ever trusts them again. Meanwhile Maxtor put out crap drives year after year and nobody seemed to mind. It was a very weird situation.

FWIW however I bought some HGSTs a couple of years ago that are by far the loudest drives I've ever used. They've been reliable thus far, but the noise level is so bad that I had to move them away from where the people are working.

Thinking of Maxtor gives me warm fuzzy feelings. That 20gb disk felt like it could store the world.

And nowadays it'll barely store the operating system (unless you make a conscious effort to strip things down).

I had one of the first double-height 70 MB drives when it came out.

It literally could hold all of the popular software of the time, around 50 applications.

I also had one of the first CD-ROM drives at work (the Japanese director had connections at Sony.)

We were like, "How can anybody possibly ever fill 650 MB of data storage?"

We ended up mastering our own satellite data CD-ROMs to demonstrate to people what was possible.

Meanwhile NTSC is sitting over there needing around a megabyte per second to store.

Pictures and movies have always been the easy answer for how it's possible to fill up huge amounts of storage.

But user behavior data is a much bigger fraction of total storage use than video.

Source?

I guess they are same design as Toshiba DT01ACA series. My 10-years old Barracuda was barely audible drowning in fan noises, but this new Toshiba seeking is obnoxiously loud.

Yeah, it would have been nice to know how loud the HGST would try to be before I bought one. But stuffing it in an isolating bracket like this basically fixed that: https://www.amazon.com/gp/product/B07LCD2RJY/

I loved and owned quite a few 2.5" HGST travelstars. There were capably of takig insane abuse and worked just fine. Interestingly enough I never got to buy the 1.5TB ones; by the time I needed one it disappeared off the market, the website, everywhere. I never knew why.

    192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       2502492302
    193 Load_Cycle_Count        0x0012   086   086   000    Old_age   Always       -       142654

^^^ those are a working, happy disk; all due to a "feature" that was on in Ubuntu by default that wanted to idle/spin down the disk every other second and it took me a while to notice.

On contrary I'd worn a 2TB 2.5" Seagate out in 2 years. Bought it new, put it in the same machine the 1TB HGST had been running for 3 years already, the Seagate died earlier - or at least that's what ZFS told me. It's spinning, but it throws weird errors from time to time.

I never had experience with 3.5" Hitachis.

Eons ago there were some interesting disks in the market though; if I remember correctly my father had some Fujitsu SCSI160 or 320 drives which had their normal working temperature around 50-60C.

Past performance does not indicate future return, but so far I'm seeing no reason not to distrust Toshiba hard drives at the very least given the Backblaze stats. I've bought 8 Toshiba drives with zero problems over several years on aggressively used drives but I've had several failures across drive families over the past 10 years with Western Digital drives of different batches and lines.

One of the dangers of blending drives is that there are sometimes drive geometry incompatibilities that can screw up RAID. Saw someone show they had drives that are a few cylinders off and can't replace drives now with certain replacements.

> Past performance does not indicate future return, but so far I'm seeing no reason not to distrust Toshiba hard drives at the very least given the Backblaze stats. I've bought 8 Toshiba drives with zero problems over several years on aggressively used drives but I've had several failures across drive families over the past 10 years with Western Digital drives of different batches and lines.

Are there too many negatives there? It seems that you first say that you distrust ("no reason not to distrust") Toshiba drives, but then you say that your experience with them has involved no problems.

Should probably drop the dis- , thanks. Too many conditions in a row too fast is how I get bugs in my code just like in my written sentences :P

> Personally, I like to make RAID mirrors with drives of different brands, or at least different models. Even it it is not ideal for performance, it helps making sure that not all drives fail at the same time.

This is exactly my approach lately, and it works great for RAID10 arrays (where you only need 2 or 3 different vendors - one per side of the mirror - to mitigate shared bathtub curves). I usually go with SSDs nowadays, though (half Crucial, half Kingston), both for performance reasons and because they rebuild faster (and also because the limited lifetime of flash memory makes some sort of RAID essential for longevity, though this has been less of a problem with modern flash tech); if I really needed the capacity of magnetic media I'd probably opt for a WD / Toshiba split.

What do you recommend for Crucial and Kingston SSDs?

I’m familiar with Crucial BX500 and Kingston UV500. They are both SATA and are decent performance considering the limitations of SATA 3. If price isn’t a factor or performance is important I’d go with a Samsung 860 EVO or Pro on SATA. I’ve also used a smaller number of SanDisk by WD SATA SSDs and they work fine as well. Had a bad Samsung 840 series but I think failure to implement proper overprovisioning was to blame. Also had a bad Intel SATA SSD but it was an older generation product and had a small sample size of only a few Intel SSDs, so not writing Intel off; yet it’s suffice to say their products don’t seem price competitive with Samsung’s offerings.

As for NVME SSDs I have used and installed probably ten so far but I think every single one so far has been a Samsung 9xx m.2 and I’ve been extremely satisfied with all of them, especially the 960 series EVO and Pro and moreso the current gen 970 EVO Plus and Pro.

Honestly I just go with whatever's cheapest for a given target capacity (which, judging by my Amazon and Newegg purchase history, would be the Crucial MX500 and Kingston V300). Any SSD's almost always "fast enough" for my purposes (even with SATA instead of NVMe), and Kingston and Crucial have both been consistently reliable enough that I haven't felt the need to splurge much (and if they do fail... well, that's what RAID's for, right?).

Both the desktop on which I'm typing this comment (my personal "gaming" rig) and my work laptop (a Thinkpad T470) are using Crucial MX500s (1TB, M.2); the former has a couple extra M.2 slots, so I'm considering migrating to a Crucial P1 + Kingston A2000 combo (thus biting the bullet and switching to NVMe, and getting RAID going on this machine). Most of my 2.5" purchases tend to be Kingston V300s unless the MX500 happens to be cheaper (e.g. most recently for an old XP computer I'm rebuilding for a family friend) or I'm specifically buying both for the aforementioned purpose of RAID diversification (which is a bit tricky, since Crucial and Kingston use different increments for capacity, but I'm also perfectly willing to pull a bit of a SSD "no-no" and use the leftover space on whichever drive for a swap partition, or else just let the space sit unused, or perhaps use it for temporary files where I don't care about any sort of data preservation).

In any case, no complaints with any of 'em. I haven't tried the P1 or A2000 yet (let alone Kingston's pricier KC2000), so I can't attest to those, but the MX500 and V300 have both proven to be reliable workhorses.

I'll definitely be sure to try Samsung again at some point; the first SSD I ever bought was a Samsung (at Fry's), and I had pretty awful experiences with it, but that was back when SSDs were a pretty new concept so I'm guessing things have generally improved.

I think Seagate has always been low on reliability; the ST3000DM001 is an outlier but I've heard far more reports of data loss from them than any other brand. The number of threads in HDD recovery forums that refer to a particular brand makes for an interesting analysis... and even before that outlier they were below WD. From what I've seen, WD is somewhere in the middle.

My beard must be grayer than yours. In the 1990s and early 2000s, Seagate was the gold-standard for reliability, and they were priced accordingly. We literally bought nothing besides Cheetahs for our data center for more than a decade. So did everyone else.

> based on backblaze data

I don't get that at all.

I've been in the WD camp since the 90s but Backblaze's data has always ranked them around "better than the worst" and slowly eroding my confidence in them.

For me, Backblaze's Stats have changed my perception of HGST who I've held in low regard since they were IBM with the whole Deskstar issue.

The deathstars were amazing.

The 60Gb ones that everybody screamed about worked perfectly if you only partitioned them out to 58Gb.

Best value for money on the market at the time so long as you remembered you had a 58Gb hard drive and behaved accordingly.

Can you elaborate on that? I can't find anything else about this.

They were well known for failing.

The ones I got only failed on the outer edge.

Worked out nicely cost-effectiveness wise if you didn't trigger the failure mode.

The ridiculous thing is that nobody would have batted an eye if they did this with Green/Blue drives, especially if they just owned it upfront. I still don't particularly get the difference between Red/Green, or even Red/Blue for that matter, apart from opaque firmware tweaks about power saving, and some hand waving about chassis vibration. SMR/PMR would have been an obvious partition.

I still can't fathom how companies can be so oblivious about their actual customer market. People speccing individual hard drives are integrating their own systems and want to care about the details, not just some colorful indicator of "good/better/best".

You're talking about an industry that has no problem shipping known-unreliable drives for decades at a time.

They likely rebate / warranty their important hyperscale cloud, enterprise, & OEM customers. According to whatever custom contract they've negotiated.

And don't really care about the retail segment, given the margins. Most people probably won't notice. And those that do won't be able to do anything.

They apparently still care enough to create all those Red/Green/Black/Blue/Purple/Gold lines. Even if those are really just a few different #defines and different ink in the label printer, they're still additional SKUs. Also the margins seem plenty high given you can shuck 8TB Easystores for $130, with the retail version at ~$200.

And yeah individuals can never really do anything in the short term. Eventually there will be a suite of best practices to test a new drive to make sure it's not SMR, and likely some tweaks to filesystems to ratelimit rebuilds, based on knowing which drives are actually SMR. But they're arbitraging away brand loyalty to WD Red, for a short gain. Given the goodwill they enjoy(ed) from that Blackblaze study creating a refrain of "don't buy Seagate", this just seems foolish.

No file system should be doing random writes, and when you have a cluster, there is always an open stream you can append to. I agree this is a bogus move, but tech with analogous limitations but also similar enhancements will make it into more things.

If risk and failure were spread more evenly, systems would behave more like gas and less like a ceramic.

You expect a networked file system accessed by multiple users to not have random writes?

For some value of expect, yes.

> No file system should be doing random writes,

A typical size for a SMR zone is 256MB, and any writes that aren't appending sequentially to what's already been written to that zone count as "random writes" on a SMR drive. That's an absolutely massive block size to expect filesystems to work around, and I doubt there are any commonly used filesystems that can store their metadata in a SMR-friendly way.

Metadata belongs in persistent storage (PMEM or flash), file contents can striped onto a magnetic disk. Flash suffers from the same problem. Rewriting a file should be appending binary diffs to a log and reapply the writes against the first version into a new location on the cluster.

SMR is extremely painful for anyone except those that can handle distributed storage systems. I think the only place where SMR makes sense for home users is in a DVR, or game assets where a virtual disk is streamed down from a cloud provider in a single pass, and no interior mutations.

Ok, so when you say "no file system should be doing random writes", you should make it clear that you're not trying to describe file systems that actually exist in any sort of production-ready state. I would generally agree with a statement that new filesystems should be designed to avoid random writes except to solid state media. But for users wanting to deploy a real filesystem now to commodity hardware, the recommendation has to be "avoid SMR", not "use a smarter FS and tiered storage hardware".

Correct. SMR should be sold to tiered storage providers OR SMR drives should include a scratch flash or pmem device to hide the large write amplification of rewriting a block (ala back to tiered storage).

HN is fringe, I am not making statements about people with drobos.

For many copy-on-write or log-structured filesystems it's not particularly hard to adapt them to such a block size, but I don't think it has actually been done on a major scale yet.

The product lines were driven by the fact that margins were low: attempting to decommodify via artificial use segmentation.

Shucking has always had weird economics. I think mostly because it's used by the manufacturers as a release-value for excess capacity, in an attempt to somewhat avoid the memory chip boom-bust cycle.

I though the red drives had more physical vibration dampening (rubber?) than the other ones.

Looking at Backblaze I would recommend only Hitachi and Toshiba drives ...

Western Digital bought HGST in 2012, started folding them into their business in 2015, and phased out the HGST brand in 2018.

I worked for Hitachi during that time period and that is mostly true. Hitachi actually sold the division to WD in exchange for private ownership shares of WD with insane leverage because WD had most of their factories flooded in 2011.

WD Drives were actually HGST drives with a different label for several years.

I have stopped buying WDS and Seagate (their 3tb drives failed sequentially in my zraid in 4 months and I just barely replaced them with new). And I am not buying even brands that were merged with them (here is picture how hdd brands "consolidated" https://www.mydatarecoverylab.com/wp-content/uploads/2018/07...). Those companiea must be punished (by not buying their products) for all the mischief they have done to our data for the sake of maximising revenue, they need to be aware it will hurt their bussiness.

Now I am having 3xToshiba drives and one Hitachi and for now I am happy.

Does the revelation that they use SMR somehow change their reliability or performance ?

I fail to see how it makes the drives any less useful only because it is revealed that it is SMR and not PMR.

It is models from 2TB to 6TB, meaning pretty much all NAS drives purchased in the past decade. I don't know about yours, by my drives perform just as well as they did yesterday.

> Does the revelation that they use SMR somehow change their reliability or performance ?

Yes, as plainly stated in the article

> much all NAS drives purchased in the past decade

These are new models that have been introduced in the past year. You may not (probably don't) have any of them.

> It is models from 2TB to 6TB, meaning pretty much all NAS drives purchased in the past decade.

1. SMR didn’t really exist in the market a decade ago, definitely not for non-enterprise grade NAS drives

2. This is a fairly recent change to use DM-SMR and not explicitly market it as such

3. Your drive performance needs != to everyone else buying NAS drives needs.

The article implies that's only recent revisions of the 2-6 TB products that use SMR, noting issues with "the latest iteration of WD REDs (WDx0EFAX replacing WDx0EFRX)"

> Does the revelation that they use SMR somehow change their reliability or performance ?

As I clearly stated: my concern is about brand damage.

Seeing how brittle the SMR Seagate drives are (especially the rosewood family), I'm a little bit concerned.

>Shingled media recording (SMR) disk drives take advantage of disk write tracks being wider than read tracks to partially overlap write tracks and so enable more tracks to be written to a disk platter. This means more data can be stored on a shingled disk than an ordinary drive.

>However, SMR drives are not intended for random write IO use cases because the write performance is much slower than with a non-SMR drive. Therefore they are not recommended for NAS use cases featuring significant random write workloads.

[...]

>We brought all these points to Western Digital’s attention and a spokesperson told us:

[...]

>"In device-managed SMR HDDs, the drive does its internal data management during idle times. In a typical small business/home NAS environment, workloads tend to be bursty in nature, leaving sufficient idle time for garbage collection and other maintenance operations."

That seems like a pretty notable difference to me. You probably want to know that ahead of buying the drive.

> Therefore they are not recommended for NAS use cases featuring significant random write workloads.

Emphasis mine: a NAS featuring "significant random write"s, is not the usual NAS, and might not actually be best described as a NAS at all.

In WD's terms, they'd tell you that if you've got e.g. your /var partition mounted over SMB to your NAS, then your NAS is receiving a Desktop workload, and so you should use their Desktop drives in it, not NAS drives. (Consider: the drives backing Amazon EBS—which receive Server workloads—are certainly not "archival media" disks, despite being nominally "network-attached storage" disks.)

The use-case for WD Red (NAS) drives is online archival media storage. Like S3 without the high availability. In other words, the thing most people buy a NAS for: backups and home Plex hosting.

This is why, I assume, WD don't literally call these drives "WD NAS", but instead just call them "WD Red." Just because you put them in a NAS, doesn't mean they'll function well there. You have to be using them for an idiomatic NAS workload. (Which also means that WD Red drives fare just fine in a desktop or server, too, if you're using them there purely for an idiomatic NAS workload.)

How much of the NAS market is "media storage" and how much of it is "file server in a small/medium business"?

About 80:20 by units sold, I would say; leaning slightly more toward the latter by volume of drives sold, by the fact that businesses buy the larger NASes that take more drives in them, because they're actually somewhat serious about RAID recovery.

But you've got to further subdivide the use-case: a file server for a business is still not necessarily doing "random writes." You can create and replace as many small files on a filesystem as you like, and in a modern filesystem, you'll only be doing large batched writes to the disks themselves. It's only when you're writing within existing files—i.e. writing to a database of some kind—that you see a high number of random-write IOPS hitting the disk.

Mind you, there are a lot of things that turn out to have databases in them. A Chrome profile has a couple of SQLite databases in it. An iTunes library contains a database. Etc. These are the things that—if you stuck them on your NAS, and then pointed the relevant program at them from your PC—would constitute the NAS performing a "Desktop" workload. And, obviously, if you run Postgres on your NAS—or on a system that mounts a NAS disk over iSCSI—then that disk is doing a "Server" workload.

But the average business's use of e.g. shared Excel worksheets, does not "heavy random writes" make. Document/productivity software is almost always of the "on save, do a streaming overwrite of the whole file on the remote end" variety—which, again, is not a random-write workload, since the ftruncate(2) at the beginning allows the filesystem to grab a new free extent for the overwrite to write into.

Most software that knows that it can be used in a shared-remote-disk setting is built to accommodate that shared-remote-disk setting by attempting to minimize random writes (since this kind of workload has always been slow on the kind of storage used in NASes and SANs, especially when your filesystem or software RAID does checksumming.) These days, it's only the software that absolutely can't help it—that needs random writes to be performant at all—that still does them. And it's pretty obvious, to anyone using such software, that they're not using "NAS friendly" software. Mostly because it's so dang slow!

>In WD's terms, they'd tell you that if you've got e.g. your /var partition mounted over SMB to your NAS, then your NAS is receiving a Desktop workload, and so you should use their Desktop drives in it, not NAS drives.

That's still obfuscation bordering on scam IMO, if the drives have technical limitations they should be spelled out, not hidden behind broad terms like "NAS drive", especially when they're branded the same way as older drives that didn't have these limitations.

Also, you'll have to explain to me how having these drives perform correctly while rebuilding a RAID array is not a "usual NAS" scenario. Because reading TFA it seems like getting these drives in a NAS to begin with is a challenge on its own.

Yeah, that seems pretty scummy to be. I was under the impression that "NAS" drives were objectively better than "Desktop" drives.

The clerk at the computer shop tried to upsell me on the NAS drives, chortled when I said "the "I" in RAID stands for "Inexpensive" and told me that I would regret my purchase.

It seems to me that my cheapness inadvertently paid off?

That's all well and good, but, given a sufficiently large array, it's a matter of when, not if, you'll need to recover from a failed disk, rebuilding a RAID array is precisely the expected sort of workload for a NAS — and a workload that reports indicate these drives fail at.

A rebuild is not a random write workload and is written in big chunks so they should hold up still.

Except they don't, because apparently SMR is slower to write than conventional recording, so the device buffers writes to conventional tracks, then uses idle-time to re-write the buffered data to SMR tracks.

But during a re-build there is no idle time...

Surely the use cases for NASs are pretty varied - they're general purpose storage.

>Does the revelation that they use SMR somehow change their reliability or performance ?

Yes, they do not perform anywhere like PMR on writing.

It's so bad they simply should _never_ be used on a RAID or Zpool, where they have such pathological behaviour on resilvers that can easily become the trigger for data loss.

I would never buy one of these voluntarily, as they'd cause me headaches.

So, definitely, I need to know whether what I am buying is SMR or PMR. And wd's behaviour is unacceptable.

I wonder if this is why my unraid box has been consistently failing a parity check since I replaced some drives.

> I don't know about yours, by my drives perform just as well as they did yesterday.

My WD Reds do too, but that's because mine are two years old. The switch to SMR apparently only happened in the most recent model (WDx0EFAX).

Yes, they perform worse, but there is a bigger issue. They lie. The drives report themselves as being PMR when they should be reporting themselves as SMR. They actually lie about their technology type in the firmware.

Reply on news.ycombinator.com