WD Sets the Record Straight: Lists All Drives That Use Slower SMR Tech
tomshardware.com
tomshardware.com
I wonder where they keep the voices of common sense at most companies- the ones that realize a company is stronger in the long term if they are upfront with their customers rather than playing games with misdirection and delay tactics.
If we want to change this, then we need to change how control of companies is determined in our economy.
Apple is an interesting one, in that it fired Job’s. Only to rehire and let the vision go to work. Even if it takes a few years.I think that’s the exception though, not the rule.
You just gave me an aneurysm.
I was in a meeting once where some new support / warranty policies were announced. Now this wasn't a decision making meeting so I've no idea what happened there. The policy change was a straight up violation of some contracts we had with some customers. You didn't have to understand law or anything, it was pretty obvious. I spoke up if only out of curiosity to see what would happen.
Everyone with any level of power in the room seemed convinced that it couldn't be illegal, because the folks higher up the chain made that decision....
Everyone had delegated common sense to the folks upstairs.
Often when I hear of these decision I think "Someone had to say something at a meeting right?" but actually, I suspect that isn't the case.
Epilogue: Company got sued by customers first, then a state AG got in on the action, then the feds came after them. The law firm hired by the company actually quit on them before they finally settled.
Concrete example: The compliance department in a different country mandates that all passwords must be salted (great) and use modern hash algorithms from a whitelist. That whitelist consisted mostly they sha2 and ripemd family. We were using bcrypt to hash passwords, which is an actual password hashing algorithm instead of a general-purpose hashing algorithm. Since we were organizationally so far removed from the compliance department even escalating the issue several management levels had no effect other than being told "compliance is important, don't waste time on meetings about this, just change it" and "this is not a hill to die on". So we worked around this by using scrypt instead (which is not in the whitelist but based on a hash in the whitelist) but the asinine policy is still in place because it was considered too much effort to get them to change it. Of course the next team that wants to implement a web service with passwords will face the very same issue and may take them by their word and use SHA2 directly. Great success.
Anyway, this is all I wanted from them. "Here are what technologies each drive uses, and we'll be more transparent in the future."
Now I can be confident in what I'm getting when I buy a Western Digital drive. Good for them!
They should just state it clearly in the drive's specs / model ID.
> We will update our marketing materials, as well as provide more information about SMR technology, including benchmarks and ideal use cases.
I interpreted that to mean "Future SMR drives will disclose their use of the technology". We'll of course have to see whether follow through, but for now they're doing the right things.
From my perspective as a consumer, WD has turned a disaster into an advantage. I now know that WD drives will disclose which technology they use—just like I know that Apple will bend over backwards to not secretly throttle battery performance. I can't say as much for other manufacturers.
I disagree. They explicitly denied it for months until the tech press started reporting on it. They showed nothing but contempt for their customers.
Edit: I’m never going to forget this: “Well the higher team contacted me back and informed me that the information I requested about whether or not the WD60EFAX was a SMR or PMR would not be provided to me. They said that information is not disclosed to consumers. LOL. WOW.“
From: https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin...
the NAS, aka IronWolf, drives are all CMR, have always been CMR, and according to seagate will always be CMR
https://arstechnica.com/information-technology/2020/04/seaga...
As I mentioned in another comment, equally pissed WD did this on (albiet smallest size) a Black label drive. WTF would they do this on their "high performance" segment is beyond me, and completely unreasonable.
Of course both companies are on my never buy list now, but some people have lower standards.
"Reliable
Designed to operate in the always-on environment of a NAS or RAID configuration"
So right now, today, they are steering people to buy these drives for an application that they are 100% not fit for. To me, that's still a disaster.
I tend to think the revealing of dishonesty has negative externalities as a general rule. Even though of course we would rather know than not.
From my point of view, by doing the wrong thing initially, and then (especially) by persisting in it for so long, they've just turned a disaster into a bigger disaster.
I haven't walked away from this feeling like WD is fundamentally honest and open and can be trusted to disclose critical information without prompting; I walked away feeling like WD is fundamentally dishonest, and will lie even after being caught. "They stopped lying eventually!" is not a strong defense.
> I now know that WD drives will disclose which technology they use
Do you? I certainly don't. All I know is they have made, under extreme pressure, a once-off disclosure of unknown accuracy around their current drive models.
Sorry, SMR HDDs are junk. The sooner the industry realizes that they're not suitable for any purpose, the better. Having a device where 4KB writes sometimes take seconds to complete is just plain unacceptable in 2020.
What if you have data you want to write once and then store for an extended period?
What if you're just on an ultra low budget?
For those wondering why SMR drives are so bad, consider the following: SMR drives basically make the same trade-off that flash makes, but with a magnetic media that has a significant seek time and bandwidth penalty compared to flash. Any time a non-sequential write is made to an SMR drive, the whole "block" (which can be many megabytes in size) needs to be read from the disk into memory, written out somewhere else on the disk after which you can finally perform the equivalent of a flash sector erase and start writing data out sequentially again. The catch is that your hard drive can only perform sequential i/o at maybe 200MB/s vs the 3GB/s flash can. Flash sectors are also measured in kilobytes (128KB is a common size), while SMR "sectors" can end up being 16-128MB in size. Do the math. The latency is a disaster as we're talking hundreds to thousands of milliseconds of latency to do an erase. Flash can erase a sector in a millisecond, and you can have multiple erases occurring in parallel across multiple planes and dies. An SMR drive can have exactly 1 erase operation in flight at a time.
If SMR drives were of any value, you'd see them at the high end of the capacity spectrum. It's funny how those products don't seem to exist.
Now, that's a data center, but don't consumers sometimes have similar cold-storage use cases? As long as they know what they're buying?
[1] https://dropbox.tech/infrastructure/smr-what-we-learned-in-o...
Anandtech has a post based off of WD's press release from December 23rd 2019 here https://www.anandtech.com/show/15457/western-digital-roadmap... . The energy assisted MAMR drive is 18TB in capacity, while the SMR drive is 20TB in capacity. Where's the 25% capacity benefit? Shouldn't the SMR drive offer a capacity of 22.5TB?
Have a look at this https://www.anandtech.com/show/15457/western-digital-roadmap... article at Anandtech posted January 31st 2020. Note the side by side comparisons of projected growth of SMR drive capacity shipments. SMR growth projections are always right around the corner in the next 1-2 years. Why is this? Quite simply the market doesn't want SMR. Non-SMR technology continues to improve, eliminating SMR's capacity advantage after a period of time without SMR's orders of magnitude performance penalties.
If SMR was more than just a one time 25% capacity advantage it might be worth buying into. But it doesn't. It's a technology in search of a market that just doesn't exist because of all the downsides. HAMR and MAMR are the promising technologies to increase HDD capacity, not SMR.
Are you confident that you'll get a drive that can work in a NAS when on the very same blog page they claim "WD Red HDDs are ideal for home and small businesses using NAS systems" and, indeed, on the data sheet[1], they claim that WD Red drives are "the hard drive of choice for 1 to 8 bay systems"?
At the very least they need to fix the data sheet; that's where I go to assess a drive's capabilities, not a vague blog post.
[1] https://documents.westerndigital.com/content/dam/doc-library...
Just because they're no longer going to lie about this specific thing because they got enough bad press about it to force them to be honest doesn't mean they deserve any respect.
Weber, Ralph O (April 23, 2020). "SBC-5, ZBC-2: Obsolete the ZONED field". <https://www.t10.org/cgi-bin/ac.pl?t=d&f=20-054r0.pdf > (registration needed)
As a result the write speed of the drive is even faster than a non-SMR drive until the cache area fills up. If your workload does not try to write gigabytes of random junk without giving the drive any idle time, it will work great.
I'm interested if there's a benchmark showing that?
edit: or bring down some large CAD files from the server to work on locally...
The fact is, it's labelled a performance drive, much like the red drives... SMR should not even be an option here... sure add the extra smart caching/remapping, but you still want performance for larger failes.
I think there were 1TB 7mm CMR drives available as far as 5-6 years ago, although I'm not sure about the 7mm part.
Regarding your side point, there existed 320GB, 750GB and 1.5TB hard drives, so 1.5TB or even a 1.75TB would not be a weird size for a sku.
If someone tries to steal your car and gets caught, then says they only meant to steal your TV, and then when challenged says they understand that they should have been more upfront with you that they were going to try to harm you, they are not a good person. Sadly, I don't perceive any more legitimate remorse or sincerity about fixing the root cause here than there was last time.
The fact is that right now, several days after this all kicked off, if you go to WD's site and choose Products and then NAS internal drives from the menu, it still links to Reds and says (in very big letters, repeatedly) things like:
"Designed and Optimized for NAS Compatibility"
and
"WD Red Provides Storage Compatible with Leading NAS Systems"
Based on that, it appears that they are still selling drives that are not fit for their stated purpose. Everything else is just spin.
It's not perfect, but it's a compromise. People who care get new drives; WD pays a cost for their mistake.
> If you have purchased a drive, please call our customer care if you are experiencing performance or any other technical issues. We will have options for you. We are here to help.
If they were here to help they would have done so already. It’s just a distraction.
Expecting them to proactively contact customers and to recall all existing drives seems like an unreasonable ask.
I think the point is that in this case, WD have already given us plenty of reason to believe otherwise. There isn't much doubt left now to give them its benefit.
As I said above, I don't think it's reasonable to ask WD to proactively contact everyone who bought one of these drives, so a message on their official corporate blog which reads "please reach out to customer care if you're experiencing performance problems" seems like exactly the right action to take.
So, that's the lens I'm looking through: what I'm seeing, versus what would I expect to see from a company reacting appropriately. Right now, they match.
Now, if WD had a history of posting such explicit, official messages publicly and then refusing to help customers privately, I would no longer be willing to give WD the benefit of the doubt. As far as I'm aware, we're not there yet.
I would present Facebook as an example of a company that has lost this trust. I unfortunately can no believe a single thing they say in any capacity. They have issued far, far too many apologies.
I'm not sure there is only one good answer to that, but I'm quite sure that any good answer would involve not still having the broken product and misleading presentation all over the WD website right now.
I think WD ought to offer some compensation for everyone who bought drives with worse performance than advertised. Likely they will be sued and be forced to pay something like that anyways.
After I asked to be transferred to someone else, the representative told me that "WD has no official answer yet and to call again in a day or two", I pointed out the blog post that specifically directs customers to call their hotline, but then I got back into the loop of "WD has no official answer yet and to call again in a day or two"....
Product safety recalls don't require you to get food poisoning or take shrapnel in your face from a bad airbag before fixing the issue. I don't want to wait until my RAID fails a rebuild and I lose data.
All I see is a company playing fast and loose with the specs, and their marketing people not understanding why SMR would not be fit for purpose. I would bet WD doesn't have a US-based tech team making those low-level decisions. They probably outsource all of it to their contract manufacturers that make the actual drives. And since the CM isn't responsible for marketing, they were like "you can save $5.00 per drive if you use this new recording technology." The marketing people have no idea what the trade offs are and the CM doesn't care.
Then 2 years later when the first batch of these drives start failing, the marketing person who made the original decision isn't there, the CM doesn't have any record of any of the conversations, and nobody has any idea what's going on. So in a bid to try to manage the conversation without actually knowing anything their marketing people release that first blog article.
Then the customer outcry happens, people threaten to pull blanket POs, and finally the marketer realizes they have to do some real work because it's not just the angry grognard brigade making noise.
Most modern companies that sell commodity electronics work this way.
I think "left some information out of their datasheet that is relevant" is being rather kind here. They were advertising these products for one specific purpose, and the change in spec makes them unsuitable for that purpose. They then tried to upsell to a more expensive product that retained the original spec and was still presumably suitable for that purpose in their initial non-apology apology.
I understand the mechanics of how this can happen, but the fact is that this isn't the customer's fault. It's a failure of basic management and leadership skills, and criticising the company for that is entirely fair.
"WESTERN DIGITAL DOES NOT PROVIDE ANY OTHER WARRANTIES OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OR CONDITIONS OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE"
Might grab me a couple of Purples as they're more available locally to me than Seagate's IronWolf CMR NAS drives.
On the other hand, the Black with SMR is just shocking, considering that's their "performance" line.
This seems more like a misunderstanding between the drive and the OS in that the drive doesn't tell the OS it'll need a break between write batches and an OS that tries to write to the disk in a way that's not compatible with the device technology.
The same more or less applies to SSDs - the drive can't write a disk block, but only a memory block at once.
Had I a stack of them, I would have considered the black line safe for home NAS. Apparently not, now that I know.
Lots of recording systems won't record unless the camera sees motion. Or they record at a much lower frame rate until motion is detected & then go into high speed mode to capture fine details. So that helps, and the CMR buffer also helps. But if the recording fills up the CMR buffer, you're back to incoming data potentially arriving faster than SMR can rewrite.
Time to drop back to 1080 HD cameras from 4k cameras maybe. Or reduce the number of cameras.
edit This is comparing a SMR to a similar CMR drive. Obviously, any drive can run into trouble if you exceed its write capabilities. But a SMR drive's write rate is going to be lower than a CMR drive, once it's buffer area is full. And especially if you're doing random writes.
uh, except surveillance usage is often raided for redundancy.
Surveillance data is recorded on top of the old one, which means all writes are rewrites and that is the worst case scenario for a SMR drive.
A host-managed drive with carefully aligned zones may avoid this issue in this scenario, but I don't know how far a drive-managed SMR drive can mitigate it.
I've tried out a couple Seagate SMR drives for NVR use. They weren't advertised as SMR but I figured it out from the observed performance and the "rated workload" on the spec sheet.
It's okay if you stay within the rated workload, but that's pretty limiting. This is given in the spec sheet in terabytes per year; if you divide by 4 you have approximately Mbps. One of my drives (ST8000DM004-2CX188) is rated for 55 TB/year, or 14 Mbps. It seems fine with 2 cameras that each have a <= 6 Mbps main stream and <= 1 Mbsp sub stream.
But if you want to record a bunch of streams to one drive (perfectly reasonable with CMR), these can't do it. When device-manager SMR is used with a standard filesystem (I'm using ext4) and non-SMR-aware software/write patterns (my software just writes new files with ~1-minute segments of H.264 and deletes old ones), the write amplification gets bad and the drive can't keep up. The software just gets stuck waiting for writes/syncs to go through. And it's really confusing to observe on the host because all the mapping and stuff is hidden from you. You just see on "iostat" that the "%util" is much higher what you'd calculate based on seek + throughput for the data you're transferring.
I imagine with host-managed SMR and SMR-aware software, this could be much better. Most NVRs mostly just record in a ring buffer—for a given stream, they almost always overwrite the oldest data. You could bypass the whole CMR cache area and not read/rewrite any adjacent areas if you do it well, and then the write amplification would go away.
Is it especially price competitive?
Are these "5 seconds", or "60 seconds", or "300+ seconds" timeouts that are triggering the rebuild failure?
SMR drives have slow random writes and paper over it with caching, until the cache gets full. Then in theory what should happen is that the actual write speed of the drive is exposed. That means resilvers would take a long time, but they should still finish.
What seems to be actually happening is that some of these drives have a firmware bug such that when caching is enabled and the cache gets full, the write speed drops to zero. The system then regards the drive as faulty and boots it out.
So it seems like this should be solvable with a firmware update that causes the drive to behave differently (slow rather than stopped) when the cache gets full.
This also implies that some other SMR drives with different firmware might not behave like this, and that it might not happen with ordinary RAID rebuilds as opposed to ZFS resilvering because then the writes should be almost entirely sequential (i.e. what SMR is good at) as opposed to ZFS which is more random.
What's probably happening is one of two things. Either the cache gets full and the drive is blocking while it flushes the cache to the disk, or the drive is advertising that it can do a large number of simultaneous write operations which it can't actually do all of in a reasonable amount of time when the cache is full, and then the ones queued last time out. In the first case they could have the firmware continue to process uncached writes when the cache is full, in the second case don't advertise the ability to do as many simultaneous writes.
Another alternative might be for the system to use a longer timeout value for these drives, but whether that's reasonable depends on how long it would actually have to be.
Don't forget the reason "NAS" drives exist in the first place is that several years ago drive manufacturers added a feature where if a read failed they would go into an extremely thorough but long (60+ second) recovery effort to get the sector back. RAID controllers would just see the drive stop responding to commands and mark it dead. So the NAS drives come with firmware that doesn't do the extreme recovery and instead just returns "read error" and lets the RAID controller rebuild it with the parity information.
If the drives go out to lunch due to a SMR writeback bottleneck then they will have lost their main selling point. Presumably in the normal case the drive will write the data just fine, but at a slower rate so you can rebuild your array but it will take all week. However, if one of the sectors fails the CRC check after the write and it has to try several times to get it I can definitely see the RAID controller getting frustrated and kicking it out.
I would be interested to see if any RAID software comes with a "SMR" mode where if a drive stops responding to commands during a rebuild the controller lets the drive take a 20 minute break before resuming the rebuild.
Hang on a sec. Is this documented somewhere?
I bought a WD Red to plug into my Raspberry Pi which I use as a file server. There's no RAID, just the one disk. I thought I was buying a more energy efficient or bulk-storage-oriented drive.
But if what you say is true, then the "NAS" or "Red" drives should _never_ be used outside of a RAID because robust error correction was removed from them by design. Do I have that right?
I'm afraid that does mean your choice of a Red for a single-disk system was not ideal. Presumably you keep backups of any valuable data anyway, but if downtime for recovery would be a significant problem for you then you might want to consider replacing that drive with something more suitable for your situation.
Since ZFS does checksum verification on every read, it has a much better chance of recovering from a few bad sectors.
Downside though is that the default RPi installs are 32bit and ZFS was written with 64bit-only in mind, and AFAIK there are still some issues and limitations when running on a 32bit system.
I do have backups, that's not my concern. My concern is that _when_ there is a read/write error (which are completely normal events with today's hard drive technology), the drive just gives up right away instead of making a few attempts. This could easily translate into (silently!) lost data in a single-disk scenario.
Basically, NAS drives have a hard limit on how long they'll try to recover from errors before just reported the failure back to the RAID controller so that it can handle them.
smartctl -l scterc,<READTIME>,<WRITETIME> /dev/xxxx
WD Red drives should retain this setting across reboots. Some drives don't, and some don't support the command.
So.. why do the RAID rebuilds have timeouts on NAS drives at all? If you paid all that extra money for a special firmware that doesn't time out on media error, and the drive is still accepting and processing commands in less than X hours per command, then wiring in your own timeout seems like a really bad idea.
When the cache is full and something sends a write to the drive, does the drive still accept "are you still there?" commands while the write is queued?
There's the problem right there! :) Drives aren't trustworthy, regardless of label.
It reads to me like the fault here isn't just on the hard drive manufacturers, like everyone's made it appear in top-level comments of both issues about it this week. I'm glad I asked more questions so that I'm better informed to help my friends when they encounter this. I appreciate everyone in the thread offering help with the details.
I'd love to see the CEO read that with a straight face.
One thing I have started doing is running NoScript but by default I allow scripts. But when a site puts up notices about adblocking or other annoying things I disable scripts for them. Surprisingly it often allows me to read the website without issues. I did not do that with Tom's Hardware though.
Announce a trade-in for anyone who has a Red and has a problem with device-managed SMR, with a CMR Red. Many, many people won't notice or care (RAID1 users who don't do massive writes) Then reflash the SMR reds with the host-managed firmware they already have for the megascalers (who of course have their own highly proprietary software management layers to handle SMR) and sell those at some deep discount to anyone who wants them.
Announce a $100,000 bounty to anyone who adds host-managed SMR support (with some slightly ambitious perf target vs CMR) to ZFS, bcachefs, whatever.
They come out of this looking good (eventually) instead of looking like complete assholes. And end up with an expanded ecosystem that allows them to create a new product family using firmware work that they've already done years go!
Which is why beyond some basic hinting its not worth putting in mainstream filesystems.
Tapes don't have fast (by spinning rust standards) random reads.
Hinting in ext4 or NTFS is of course a waste of time, but CoW filesystems like ZFS or btrfs are in a much better position to handle host-managed SMR.
And a generic "SMRTL" layer that utilizes (hopefully mirrored) SSDs as a write staging area could do massively better than the dumb host-managed SMR on these drives.
Is this true? On the assumption of the drive Firmware working correctly as intended, why would this be a problem?
I have not been following the problem closely as I dont quite understand what the fuss is about. I expect HDD to be slow, so not much of a problem to me. I only cares about its reliability. ( Which is something that theoretically SMR should be worst, there are no data to back it up yet )
The problem is how slow. 10MB/s sounds ok since it's faster than many home internet connections. But it's going to take you 3 weeks to re-build a 15TB drive in a raid array. Much longer again if the array is used at all (causing non-linear access as it has to rewrite changed blocks.)
WD added new information. Their previous post didn't say which drives actually used SMR, and took a considerably different tone.
Why WD decided to keep the same URL is anyone's guess, but it's different content now.