The Western Digital WD Black SN850 Review: A Fast PCIe 4.0 SSD

I've started to upgrade parts of my network to 10Gbps, and it was surprisingly affordable. My NAS and server now have a dedicated 10GB Gbps link between them (the NAS also has 4 1Gbps connections to my primary network), and it only cost about £50. Obviously it still could be a bottleneck, but it's a decent step in the right direction.

If 3200 is your RAM's stock speed, you can probably run it at least 10% faster using "DRAM Calculator for Ryzen".

Heatsinks like the big chunk of metal in that video are only really good for delaying thermal throttling by adding extra thermal mass. The drive will still hit 100C and still throttle if you push it hard enough. Only airflow will remove heat from the drive.

If someone has workloads that are so consistently intense that throttling is causing an impact, they probably shouldn't be using consumer drives anyway. Server grade drives are designed for sustained workloads.

When your drive can sustain 7GB/sec read speeds and 5GB/sec write speeds, you can read an entire 1TB drive in about 2.5 minutes and write the entire drive in 3.5 minutes. Throttling will kick in after a minute or two, but the bigger problem is that if you're doing this consistently you're going to exhaust the drive's 600TBW write endurance very quickly.

In theory, if throttling wasn't an issue you could consume the entire drive's write endurance in a matter of days. If you're doing that level of writing, it's time to upgrade to more appropriate hardware.

Sufficient airflow across the motherboard is also important, of course.

https://www.ebay.com/itm/20Pc-ST036Y-12-12-3MM-Aluminum-Heat...

https://www.ebay.com/itm/12pcs-14x14x6mm-Small-Anodized-Heat...

https://www.ebay.com/itm/12pcs-Small-Aluminum-Heatsink-Cooli...

In addition to non-adhesive thermal paste, everyone should have a small tube of heatsink adhesive for building modern x86-64 desktop PCs from parts.

Based on known program/erase cycle limits for the NAND they're using, SSD manufacturers can make pretty accurate projections for how long (in TB written) the drive as a whole can survive, accounting for how that drive's controller manages the flash and what kind of write amplification can be expected from the intended workloads.

The most significant limitation where manufacturers have to go out on a limb with their projections pertains to the standard requirement that a consumer SSD needs to be able to retain its data in a read-only state for a year after reaching the end of its rated write endurance. NAND manufacturers can very quickly bring flash to that level of wear, but they can't wait a year to measure retention characteristics before shipping. So they make projections based on high-temperature accelerated testing.

I think a lot of Devs have had the surprise that their laptop has higher single core performance than a server.

I don't know to what extent but space is certainly a premium. For a unit of rack space you want to maximise the cores and memory.

With a server grade multi socket CPU, you're also likely to pair it with at least 512gb of ram.

Consumer CPUs have a much lower limit. Though Threadripper is a bit higher.

There's also the availability side of things. How many CPUs can you get to populate your datacenter?

Will it be easy to order 500 consumer CPUs if you're not a retailer? Intel/AMD do want to steer their enterprise customers to higher volume parts.

Forget consumer grade: even normal Dell/HP/Supermicro stuff is a PITA to run at datacenter scale. Most major cloud providers roll their own machines for a reason.

I'm guessing any hardware savings would be evaporated by increased labor, power, and footprint costs, likely making you more expensive than other providers.

> consumer grade compute

Non-ECC compute is a no go from start. You won't believe how computers make mistakes when they're driven to their limits. Even with good cooling, frying RAM, CPUs and ethernet gear is ordinary events.

> local storage

Consumer grade HDDs are slow. SAS is not consumer grade. Consumer grade SSDs cannot handle the write cycles. The ones which can handle neither cheap nor consumer grade.

> internal networking

Gigabit ethernet is cheap but, is not fast enough anymore. 10Gig+, Infiniband and other stuff needs reliable high bandwidth PCIe, which is not in abdunance in consumer grade hardware.

You need more lanes, then we come to next point.

> Most applications that run on AWS and GCP don’t really need that premium Xeon/EPYC CPUs

Server CPUs are not about processing power all the time. Reliability, PCIe lanes, expansion and advanced virtualization which allows that high performance network to be connected in abundance and shared between VMs without performance penalties. So you can scale the number of jobs running on less hardware in smaller spaces.

> enterprise-grade storage that is 10X the cost of consumer grade drives.

Again, it's not about performance but about reliability at least half the time. Performance is required to be able to scale, not for independent jobs.

You are paying more but mainly when the capabilities are better. Example for SSD: Samsung 860 PRO 1TB and "for Data Center" Samsung PM883 1TB both have about 1200 TBW lifespan, both cost about $180.

On the other hand 8-core Xeon 4215 is almost 2 times more expensive than 8-core consumer CPU but it has 3x the memory bandwidth (6 channel controller vs 2 channel), 1TB of RAM per processor vs 128GB, 48 PCIe lanes vs 20 and supports 2-way SMP. That means you can put few times more capacity (cores, memory, in 1U rack enclosure than with consumer hardware. That additional capacity can also share 1 PSU, 1 network card.

I'm not even talking about ECC memory and remote management capabilities available only in server grade hardware.

> (let alone SATA)

NVMe is (very) noticeably faster than SATA, especially for random IO, but also for sequential io patterns. Multiple queues and reduced protocol latency go a long way. SATA III throughput maxes out at 600 MB/s. In comparison, these drives do 4000-5000 MB/s.

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.19    0.00    1.96    1.21    0.00   96.64
    
    Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
    nvme0n1       4180.00 332544.00     0.00   0.00    0.21    79.56 2518.00 1332796.00    74.00   2.85    6.89   529.31    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   18.21  81.60

905P is $598 for 480GB.

SN850 is $119 for 500GB.

https://www.anandtech.com/show/16558/micron-abandons-3d-xpoi...

Intel will need to bring Optane production in-house or get another partner on board if they're going to continue with it

> In theory, the security guarantees offered by hardware encryption are similar to or better than software implementations. In reality, we found that many models using hardware encryption have critical security weaknesses due to specification, design, and implementation issues. For many models, these security weaknesses allow for complete recovery of the data without knowledge of any secret (such as the password). BitLocker, the encryption software built into Microsoft Windows will rely exclusively on hardware full-disk encryption if the SSD advertises support for it. Thus, for these drives, data protected by BitLocker is also compromised.

Source: Self-Encrypting Deception: Weaknesses in the Encryption of Solid State Drives (https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=883...)

( And still no roadmap of Affordable 10Gbps Ethernet, even 5Gbps would have been acceptable. But looks like we will only get 2.5Gbps as replacement. )

edit: what where they thinking https://www.storagereview.com/wp-content/uploads/2020/11/Sto...

> We need to start expressing write endurance as a multiple of write speed

I can't tell if you're joking or not, but to be safe: no, we don't need to do that because there are virtually no realistic use cases where you'd continuously override data that you just stored on your persistent storage.

Honest question, this is (practically speaking) my impression after 20 years.

See https://www.anandtech.com/show/16458/2021-ssd-benchmark-suit... for more details. (Discussed at https://news.ycombinator.com/item?id=25994051 )

That's >2x higher throughput than an Optane at equivalent queue depth and (as far as I can see in the UK) at less than a tenth of the price: https://www.scan.co.uk/products/2tb-wd-black-sn850-m2-2280-p... vs https://www.scan.co.uk/products/15tb-intel-optane-dc-p4800x-...

And that's 7 GB/s from _one_ SSD. Aggregate memory bandwidth on something like the Zen3 is roughly 40 GB/s. These are also first generation PCIe 4, plenty more to come.

Doesn't require a huge improvement before you end up in a position where you simply don't have the memory bandwidth or cycles to deal with more than one drive.

I suspect the Optane wins most of the benchmarks because of it's outrageously good low queue depth random read performance - that's very effective for software that's not written for modern NVMe SSDs which benefit from very high queue depths. Check out the 4k random read performance from the SN850 at high queue depths:

https://images.anandtech.com/doci/16505/rr-s-sn850-1000.png

If you can keep the queues deep, it manages to beat the throughput of the Optane. You've got to design algorithms and data structures to exploit that kind of concurrency though.

TL;DR: SSDs are harder to kill than they might seem.

The size of the dynamic SLC, when available, is roughly 1/3 of the erased space, with an upper bound.

If you don't need the capacity, set aside 30% of the drive in a partition you won't use, then use blkdiscard to erase it. Not only will this increase write performance, it may increase endurance. Increasing endurance with these drives is of interest because they are rated at 0.3 DWPD, compared to 1 DWPD with previous generation drives.

Taking over-provisioning to an extreme, I found that erasing a 980 Pro, then creating a 100 GB partition allowed me to overwrite that partition many times over at a sustained 5000 MB/s. This could be interesting for journaling, so long as you are ok with lack of power loss protection (unlikely for a journal) or cache flushes could be performed to ensure the data is on NAND, not just in DRAM.

But really, if you need to sustain writes at such a high rate, both of these are the wrong drive for endurance reasons. With optane stuck at PCIe gen 3, it may be great for low latency small blocks, but it won't exceed 3500 MB/s in a x4 NVMe form factor.