Understanding the Intel Optane Shutdown

I think one of the major missing factors is lack of OS support for the sorts of features that NVDIMM could potentially bring to the table. Super-fast wake from deep hibernation is just a starting point, the real work is re-writing every line of code written since the dawn of computing which starts from the two assumptions "one kind of storage is persistent and slow" and "the other kind of storage is volatile and fast".

HPE made noise about challenging those assumptions with The Machine but predictably lost interest as their own memristor technology lost steam.

> What could have saved it?

Persistent memory was/is a big topic in the database research world. The idea was that you would not need to worry about ACID since all your data is still there in the event of a power loss/crash. It would massively simplify database code.

However as it turns out... persistent memory doesn't buy you all that much. You still effectively need a replay log to restore the state you were in before a crash event, and you still need all the restoration code. In the end it in most systems it just behaved as a very fast (and expensive) SSD. The benefits vs. price just wasn't there as a modern DB setup on a fast SSD + more RAM would get you much more performance per dollar.

What you'd really need is transactional memory, not just persistent memory. And you'd need a database engine written completely from the ground up to use this memory. Probably several hundred KLoC of tricky code (and all the joys that come with parallel programming), targeted at hardware that doesn't even exist yet. Sounds very expensive and risky to me.

I think eventually we'll see something in this line of technology become commercially available. But I'm not surprised that Intel pulled out.

It could have made a big difference but it was adoption that was the problem. Swapping an SSD from a HDD is very easy and straight forward for most people - SATA cables are also well designed and easy to understand. Its almost like a USB cable in an L shape.

Completely changing the format? That is a huge obstacle.

NVME SSD's also came along soon enough we got something that was much, much faster than SATA SSD's and eventually I think it just became clear it was diminishing returns. That being said I am still hoping for an Optane replacement.

> One of the things that drove mass SSD adoption was the experience was plainly better. Even the most illiterate user could see how fucking fast an SSD machine booted and how much more responsive they were.

> SSD to Optane? There's no practical difference.

No, that's false. That's true of NVMe in general but Optane specifically is really good at the low-queue-depth random-4k workloads that characterize consumer use-cases.

Optane actually is extremely fast at things like OS patches that normally take a few minutes even on NVMe flash let alone sata. They boot faster, applications launch faster, etc. Normally those things don't have lots of parallel threads running so queue depth is low and flash doesn't perform very well (as in, SATA and NVMe are nearly indistinguishable), but Optane really helps.

Is that worth paying 10x as much for? No, probably not, but that's the real problem, not the general lack of performance. Optane is actually noticeable specifically for being the first thing that actually provides a noticeable improvement above and beyond SATA SSDs for consumer use-cases. It's just also too expensive to be justifiable.

They should have sold it as level 4 cache

What's more, Optane was sold as... a cache to have in front of your spinning rust. In an age where SSDs were cheap enough that you could just get a few TB of SSD and have all your data access sped up, not just what happens to be in cache.

Interesting - do you mean Optane PMem (i.e. where it is connected as a DIMM)? Or the Optane SSDs?

IIRC Intel bought Micron out of the 3DXpoint/Optane project in 2018, and (based a some hallway conversations at conferences) folks at Micron were not exactly enthusiastic about the early results from the technology or Intel's overhyping marketing years before then.

It had good performance, but part of the problem is that Intel way oversold it at the beginning. They said they would be "1000x faster, 1000x more enduring, and 10x more dense than NAND". In the end none of these promises materialized. They released these puny little drives that had a bit better latency than NVMe SSDs, that were locked to Intel systems, that OSs didn't know what to do with them, and that didn't quite fit anywhere in the memory hierarchy.

Yeah, it found a few niche uses, but it was never going to be sustainable.

> Many here seem to miss how deeply complex SSD's are and how truly different optane is from most consumer and enterprise SSD offerings. Most consumer ssd's only optimize for sequential reads and writes and at that max throughput, not latency for random access. Glossing over a lot of detail, but this is where optane truly shines.

Yep. Optane was much much closer to an "ideal" block device. There is no block write/page erase mechanism, no garbage collection, no interpolation layer between you and the hardware at all actually. Reads and writes onto a platonic block device. Latency is very low for xpoint so that's fine, on paper.

It also means, that for ZFS, it's basically fine even without the battery backup. The xpoint isn't lying to you about a committed write so there's no reason to have a caching layer to manage the physical flash pages, in the way that flash SSDs need backup power for their controllers to flush the cache.