Micron launches 60TB PCIe gen5 SSD with 12GB/s read speeds | Hacker News Reader

Micron launches 60TB PCIe gen5 SSD with 12GB/s read speeds

micron.com

142 pointsLwrless66 comments

Nice to see the product brief break down the write endurance rating by workload. They really want you to use at least 16kB block sizes with this drive. NAND page sizes have been larger than 4kB for quite a while now, but most SSDs still put significant effort into handling 4kB IO operations well. This market segment of server drives optimized for capacity over performance or endurance is where vendors are most willing to be up front about how your workload really should be appropriate for the drive.

Or it could be a boon for video/audio/graphics types. I’m used to setting up my spinning rust arrays with 64k block sizes. Continuous reads from large files vs lots of tiny files is something that tripped up a lot of storage vendors trying to break into the media world.

It's very expensive. Gen5 SSDs are around $100/TB. Hard drives are $16/TB.

It makes sense to have a small fast SSD for editing, but HDDs for mass-storage. And with video you usually want redundancy, storage gets expensive fast.

And with any data you usually want redundancy,

FTFY

But I know what you meant. I used to have systems built with multiple small arrays where the original sources were in RAID-5 (pre-raid6). The software would read from that array and write to a RAID0 array as these files were only ephemeral. The software would then read from that RAID0 and write to a second RAID0 to avoid reading/writing to the same volume. This was all before SSDs. For the lulz, I would love to see how that workflow would perform with NVMe storage.

At the time of using these systems, spinning rust was not $16/TB and was probably closer to the $100/TB. I still remember $1k/GB in the early 90s

At the time you didn't have 8K or RAW videos or 200-megapixel photos.

There's never enough storage.

While not 8k, it was 4K-5k and definitely RAW. While never enough storage, we have achieved a point where disk I/O is not a bottle neck. It used to take a raid0 7 disk stripe to record SD video only while the audio was written to a separate drive. The fact we can record 8k RAW with audio on a single drive the size of a stick of RAM is crazy to compare. Smaller faster cheaper.

Physical NAND pages aren't really relevant. At least some high end SSDs I know of would gang multiple NAND chips together anyway and effectively the page size would be X*physical (where X might be 8 or 16). It could still do 512 byte writes well if it was configured that way.

The limiting factor is the size of the forward map structures. To a lesser degree also things like garbage collection overheads and reverse map structures (particularly if you support deduplication or snapshots). But forward map is the main thing.

16kB pages is fine with me, so long as drives have Flexible Data Placement, where they can specify a placement group for their writes. This will let the drive write associated data together & avoid write-amplification problems.

Alas I have yet to see many drives advertise FDP clearly. In fact I've seen deception, such as Micron claiming to demo FDP with their 7450 Pro series, but they aren't testing official NVMe FDP, they just namespaced their drive into four equal sizes. One major advantage of FDP is it doesn't need ahead of time subdividing! https://www.micron.com/about/blog/storage/ssd/benefits-of-fl...

This was supposed to be a pretty low lift for driver makers, and the hyperscalers were beating down the door for FDP. I'm shocked it's a year latter and it's still not clear how we can get drives that will work with us to combat write amplification.

I believe they always do that with the ION line which has crap endurance for random writes (relative to the average datacenter SSD), so I assume it's a way to reassure customers that the non random is not that bad.

Pablo Escobar used to announce the price of cocaine on the radio every day, but you will not know even the estimate for an SSD :-)

SSDs cost more per gram?

printer ink sure does

Moore's law: making us disappointed by the storage industry since 1965.

There are a few retailers that carry enterprise SSDs so you don't have to "call for pricing", eg. https://www.wiredzone.com/shop/category/components-hard-driv...

For those who don’t want to click thru: the link shows Gen 4 60 TB drives from Western Digital and Solidigm for ~$7,262.00 ($121/TB). The Micron Gen 5 part is likely much more money.

Note on eBay there are currently many listings for 2018-era Intel 8TB P4510 Gen 3 drives, some for as little as $50/TB. These drives have great (enterprise) endurance and performance and became a key part in the sale from Intel to Solidigm. The Micron listing is certainly a milestone, but the NVME market is getting as crowded as CPUs (e.g. 10-year-old Haswell is still plenty useful today).

> $121/TB

Huh - that seems quite reasonable, actually.

Looking forward to have this in my homelab in 10 years.

Woohoo exactly!

In 10 years they'll all be worn out.

I'm pretty sure he means brand-new versions, with the idea that high-end enterprise tech like this will eventually become inexpensive consumer-level tech, as it usually does.

Old drives on eBay are somehow usually in great condition and barely used in my experience.

You would be surprised! Enterprise stuff get decomissioned after a set amount of time, it doesn't matter if they still work.

True. In our storage we had complete servers and storage originally packaged in case server in our Datacentar would die. After x years we decommissioned both of them. It was a heartbreaking moment to throw stuff to recycling that was not used and mint packaged. I secretly hide some Pentium I laptops that came with storage and gave them to collectors.

you aren't getting a second hand enterprise drive anyway unless it's bootlegged. it's like infosec 101.

Any modern SSD would have secure erase. Whether it's used or not is uncertain.

cheaper to mill them in bulk once a year.

SSD prices have somewhat been sticky & not coming down for some reason...

https://pricetracker.wtf/product/63777900-d7b6-4391-addf-b03...

That particular model may be a bit special in that it can be modded to run in SLC mode, and a 500GB SLC SSD for ~$100 ($0.20/GB) is great value.

TIL - thank you.

Details about this mod:

https://theoverclockingpage.com/2024/05/13/tutorial-transfor...

Why cant we get any of the new nice connectors on consumer hardware? The fact that you will have to get old enterprise hardware to get ECC and any kind of pcie drive connections apart from M2 is stupid.

> you will have to get old enterprise hardware to get ECC

not true. xeon workstation and a bunch of amd chips support it

Some non-xeon chips support ECC now too, in the 12th generation and onwards.

The issue in the consumer space is the motherboard support more than anything- even if your cpu supports ECC; your motherboard rarely if ever advertises support.

https://www.tomshardware.com/news/intel-enables-ecc-on-12th-...

Supermicro + elderly intel i3 with ecc working running a fileserver here but neither of those vendors look like good bets for the near future. Asrock + cheap epyc seems to be the most plausible replacement when it dies.

I tried that AMD thing, and getting things to work properly is hard. Hardly any motherboards advertise ecc support, and getting it to work is finicky and brittle. I stopped using it on my main computer after it silently stopped working and started giving me other memory issue after an OS update.

Skill issue. You had to do some actual research on the boards. I grabbed a few manuals and ended up with an asrock.

This was on ASRock X570M Pro4, with a memory from the qvl.

I validated that it worked, and then one day it stopped working after a system update. I gave up on ecc on AMD consumer chipsets. From what I gather it is still shaky.

I have i3-8100 from years ago that supports it and runs my NAS just fine. Supermicro boards support the chip too.

You're not a consumer if you have the need and money to buy this stuff.

I am not trusting my storage to something without ecc. It is as simple as that.

Totally agree and I'm still annoyed by at psu connectors. Why do motherboards still the giant clunky 24pin connector, it's just ridiculous.

I see they list an MTTF of 2.5 million device hours. That works out at ~285 years - is that really correct? (I'm not really familiar with what's considered normal for devices of this class)

As far as I understand it MTTF measured in device hours is meant to give expectation on how often to expect a failure when you buy a pool of devices and run them a reasonable amount of time (I.e. a number to tell you when you buy 2,000 of this how to calculate your expected failure interval) not necessarily "how long an individual device is expected to keep working".

It means if you buy 1000 of them, one might die after 3-4 months, and another one or two within the first year.

This[1] page explains MTTF as follows:

The MTTF is a statistical value that defines after how much time a first failure in a population of devices may occur (measured in hours). If MTTF is given as 1 million hours, and the drives are operated within the specifications, one drive failure per hour can be expected for a population of 1 million drives. For the more realistic quantity of 1000 drives, a Managed Service Provider (MSP) should plan for a failure every 1000 hours (almost 42 days).

It also has some formulas to convert MTTF to Annualized Failure Rate (AFR).

[1]: https://www.toshiba-storage.com/trends-technology/mttf-what-...

Apple must be rubbing their hands and licking their lips thinking about the markups from their base models.

i am really excited to wait 5 years to get one from ebay

Netflix

Where can I read about the storage drives used at Netflix?

I know there’s info on the Gbps they push, and using AMD chips, and FreeBSD.

But I haven’t read anything about their storage setup.

https://arstechnica.com/information-technology/2022/10/reddi...

The latest info I've seen: "16x Intel Gen4 x4 14TB NVMe" https://papers.freebsd.org/2022/EuroBSDCon/gallatin-The_Othe...

And then everything is throttled by single-threaded kswapd in linux which is not fast enough to reclaim memory pages..

so how much do these bad boys cost

If you have to ask…

The older, worse, smaller ones seem to run about $4k for the 30TB version just as a random data point.

https://www.newegg.com/p/1WK-0046-00133?Item=9SIA12KKAG9016

Another different product: Solidigm 61TB for $3.7k (Feb 2024)

https://www.tweaktown.com/news/95730/solidigm-61tb-monster-s...

Although $7340 here: https://www.newegg.com/solidigm-61-44tb-d5-p5336/p/N82E16820...

Honestly that’s not bad at all. It works out to ~$370 for 6TB. I’ve paid about $150 for 2TB as a consumer when building my laptop so I’d imagine the cost isn’t too bad for business customers.

Compare to what Apple charges, thats a bargain.

Hopefully that should be enough for the swap file while Chrome is running.

If your use case is intense random IO writes a pcie 3 generation optane drive would be a better choice. Significantly less space and much lower sequential thoroughput but much better latency and about the same random IO thoroughput and probably better TBWs.

Have nand ssds caught up with optane on random? Last I've heard they were still handily beating them but it has been a while

No. Nothing on the market today is faster than a 905p when it comes to latency.

Really is bizarre considering it’s an 8 year old product.

Mine died a few months ago (I'm pretty sure just the onboard electronics died, not the cells) and I miss it.

I'd heard the Micron 7450 Pro 15 TB u.3 is comparable to an Optane 905p 1.5TB u.2 when it comes to random 4K IO; spec sheet states, "Latency, READ (TYP): 80μs, WRITE (TYP): 15μs" and 500k iops. Whereas a typical Optane 905p has 10μs writes up to 500 kiops. (ref: https://www.micron.com/content/dam/micron/global/public/docu... vs https://images.anandtech.com/doci/16636/rr-rate-905p-1500.pn...)

Over that range they're comparable it's ~10 microseconds vs ~15 microseconds. Really impressive stuff for a charge trap SSD. Size for size, and cost for cost, I'd rather have optane though.

What’s new here?

A 61TB NVME was launched 18-months ago

https://www.techradar.com/pro/worlds-highest-capacity-pcie-s...

PCIe 5 is what's new

It's also TLC not QLC.

Reply on news.ycombinator.com