What makes Intel Optane stand out (2023) | Hacker News Reader

What makes Intel Optane stand out (2023)

blog.zuthof.nl

220 pointswalterbell157 comments

It stands out, because it didn't sell. Which is weird because there were some pretty big pros about using them. The latency for updating 1 byte was crazy good. Some databases or journals for something like zfs really benefited from this.

It feels like everyone figured out what to do with them and how just about when they stopped making them.

Same for the Larabee / Knights architecture. Would sure be fun to play around with a 500 core Knights CPU with a couple TB of optane for LLM inference.

Intel's got an amazing record of axing projects as soon as they've done the hard work of building an ecosystem.

> 500 core

The newest fully E-core based Xeon CPUs have reached that figure by now, at least in dual-socket configs.

Yup. And high end GPU compute now has on-package HBM like Knight's had a decade ago, and those new Intel CPUs are finally shipping with AVX reliably again. We lost a decade for workloads that would benefit from both.

But I'm surprised PCIe based CPU+RAM modules aren't a "thing" since that's basically what a GPU is if you ignore all the rather fundamental differences. Seems like it would be convenient to cheaply attach additional compute without worrying about all the other stuff.

I suppose I'm just reinventing SXM at this point. The BC-250 comes close but despite the formfactor it isn't actually a PCIe card. Although if it integrated a 100 Gbit SFP slot it might actually be superior to a solution that resided in a host system. But the BC-250 is very much an anomaly as opposed to the norm.

You need CXL to extend the cache coherency properties of actual RAM over a remote link. That's costly tech. Otherwise, you're relying on the OS (and even the compiler/basic libraries, since you need to make fences, etc. OS-visible) to paper over the differences by doing its own implementation of distributed shared memory (this is known as a 'SSI' or single-system-image approach) which has significant challenges and is closer to the spirit of setting up swap.

I didn't mean anything like that. Just the equivalent of a GPU with the ability to run arbitrary CPU oriented programs.

Of course GPUs do many tasks very well but there are also plenty of problems that aren't well suited to them. Well I suppose I've answered my own question at this point. There probably just aren't enough real world problems that aren't amenable to running on a GPU while also being either compute or memory bandwidth bound.

Still the near-monoculture does strike me as odd. I guess GPUs have bifurcated into enterprise versus consumer at this point but otherwise all we've got is a single CPU example from over a decade ago and a single alternative take on the concept from Fujitsu. Is it just due to the obscene cost of masks for modern process nodes?

Things like that existed in the category of accelerator cards. Xeon Phi (Knights) is one example, focused on core count. Some from HP have soldered on SSDs too. You also had blade servers which is more focused on that use case, though that's going out of style.

I don't think PCIe is really a good fit for general CPU tasks. You need big heatsinks and power and can't fit that much RAM on board.

When most people are running databases on AWS RDS, or on ridiculous EBS drives with insanely low throughput and latency, it makes sense to me.

There are very few applications that benefit from such low latency, and if one has to go off the standard path of easy, but slow and expensive and automatically backup up, people will pick the ease.

Having the best technology performance is not enough to have product market fit. The execution required from the side of executives at Intel is far far beyond their capability. They developed a platform and wanted others to do the work of building all the applications. Without that starting killer app, there's not enough adoption to build an ecosystem.

> There are very few applications that benefit from such low latency

Basically any RDBMS? MySQL and Postgres both benefit from high performance storage, but too many customers have moved into the cloud where you can’t get NVMe-like performance for durable storage for anything remotely close to a worthwhile price.

I'm saying that there are very few downstream applications that use databases that benefit from reducing latency beyond the slow performance of the cloud. Running your database on VMs or baremetal gives better performance, but almost no applications built on databases bother to do it.

Optane was a victim of its own hype, such as “entirely new physics”, or “as fast as RAM, but persistent”. The reality felt like a failure afterwards even though it was still revolutionary, objectively speaking.

Optane didn't sell because they focused on their weird persistent DIMM sticks, which are a nightmare for enterprise where for many ordinary purposes you want ephemeral data that disappears as soon as you cut power. Thet should have focused on making ordinary storage and solving the interconnect bandwidth and latency problems differently, such as with more up-to-date PCIe standards.

I don't think that would be my main complaint. Sticking optane in a dimm was just awkward as hell. You now have different bits of memory with very different characteristics, & you lose a ton of bandwidth.

If CXL was around at the time it would have been such a nice fit, allowing for much lower latency access.

It also seems like in spite of the bad fit, there were enough regular options drives, and they were indeed pretty incredible. Good endurance, reasonable price (and cheap as dirt if you consider that endurance/lifecycle cost!), some just fantastic performance figures. My conclusion is that alas there just aren't many people in the world who are serious about storage performance.

Can Linux differentiate that different dimms are different? Or does it see it all as one big memory space still?

Yes, Linux was aware of the difference via ACPI tables.

>Which is weird....

It isn't weird at all. I would be surprised if it ever succeed in the first place.

Cost was way too high. Intel not sharing the tech with others other than Micron. Micron wasn't committed to it either, and since unused capacity at the Fab was paid by Intel regardless they dont care. No long term solution or strategy to bring cost down. Neither Intel or Micron have a vision on this. No one wanted another Intel only tech lock in. And despite the high price, it barely made any profits per unit compared to NAND and DRAM which was at the time making historic high profits. Once the NAND and DRAM cycle went down again cost / performance on Optane wasn't as attractive. Samsung even made some form of SLC NAND that performs similar to Optane but cheaper, and even they end up stopped developing for it due to lack of interest.

Cost was fantastically cheap, if you take into account that Optane is going to live >>10x longer than a SSD.

For a lot of bulk storage, yes, you don't have frequently changing data. But for databases or caches, that are under heavy load, optane was not only far faster, but if looking at life-cycle costs, way way less.

Optane was in the market during a time when the mainstream trend in the SSD industry was all about sacrificing endurance to get higher capacity. It's been several years, and I'm not seeing a lot of regrets from folks who moved to TLC and QLC NAND, and those products are more popular than ever.

The niche that could actually make use of Optane's endurance was small and shrinking, and Intel had no roadmap to significantly improve Optane's $/GB which was unquestionably the technology's biggest weakness.

> I'm not seeing a lot of regrets from folks who moved to TLC and QLC NAND, and those products are more popular than ever.

That's interesting. Even TLC has huge limitations, but QLC is basically useless unless you use it as write-once-read-many memory.

I wish I have bought a lot of SSDs when you could still buy MLC ones.

> QLC is basically useless unless you use it as write-once-read-many memory

The market thoroughly disagrees with your stupid exaggeration. QLC is a high-volume mainstream product. It's popular in low-end consumer SSDs, where the main problem is not endurance but sustained performance (especially writing to a mostly-full drive). A Windows PC is hardly a WORM workload.

Seems like it is though? Most consumer usage does not have much churn. For things like the browser cache that do churn the total volume isn't that high.

The comparison here is database and caching workloads in the datacenter that experience high churn at an extremely high sustained volume. Many such workloads exist.

There's a very big difference between a workload where you have to take care to structure your IO to minimize writes so you don't burn out the drive, and a workload that is simply easy enough that you don't have to care about the write endurance because even the crappy drives will last for years.

Of course. The inferior but cheaper technology is more cost effective in most cases but for certain workloads that won't be the case despite being more affordable per unit upfront.

The workloads flash is more cost effective for (ie most of them) either aren't all that write heavy or alternatively leave the drive sitting idle the vast majority of the time. The typical consumer usecase is primarily reads while it mostly sits idle, with the relevant performance metrics largely determined by occasional bursts of activity.

Consumer usage does not have much churn, but the average desktop is probably doing 5-50 drive writes per year. That's far away from a heavy database load, but it's just as far away from WORM.

Write endurance of the drive would be measured in TBW, and TLC flash kept adding enough 3D layers to stay cheap enough, quickly enough, that Optane never really beat their pricing per TBW to make a practical product.

I have to wonder if it isn't usable for some kind of specialized AI workflow that would benefit from extremely low latency reads but which is isn't written often, at this point. Perhaps integrated in a GPU board.

The extra capacity of modern SSD is a good point, especially now that we have 100TB+ SSD.

But Optane still offered 100 DWPD (drive writes per day), up to 3.2TB. Thats still just so many more DWPD than flash ssd. A Kioxia CM8V for example will do 12TB at 3 DWPD. The net TBW is still 10x apart.

You can get back to high endurance with SLC drives like the Solidigm p7-p5810, but you're back down to 1.6TB and 50 DWPD, so, 1/4 the Intel P5800X endurance, and worse latencies. I highly suspect the drive model here is a homage, and in spite of being much newer and very expensive, the original is still so much better in so many ways. https://www.solidigm.com/content/solidigm/us/en/products/dat...

You also end up paying for what I assume is a circa six figure drive, if you are substituting DWPD with more capacity than you need. There's something elegant about being able to keep using your cells, versus overbuying on cells with the intent to be able to rip through them relatively quickly.

We don't care about (TBW/TB) at the consumer level, we care about (TBW/$), and 3D TLC was far, far cheaper per TB, so much so that TBW/$ was not a numerical advantage of Optane.

That left ONLY the near-RAM-read-latency, which is only highly beneficial on specific workloads. Then they didn't invest in expanding killer app software that could utilize that latency, and didn't drop prices sufficiently to make it highly competitive with big RAMdisks.

In 2018, with Optane drives launching around $1.50/GB and TLC flash drives around $0.15/GB, it wasn't that much cheaper. As far as I'm aware Optane had a lot more than 10x the endurance.

10x endurance and 10x latency reduction, for 10x the price. It was closer to 5x the price with the closeout firesale after it was killed.

Optane practical TBW endurance is way higher than that of even TLC flash, never mind QLC or PLC which is the current standard for consumer NAND hardware. It even seems to go way beyond what's stated on the spec sheet. However, while Optane excels for write-heavy workloads (not read-heavy, where NAND actually performs very well) these are also power-hungry which is a limitation for modern AI workflow.

You're conflating two things. Yes, Optane would survive more writes. But it wouldn't survive more TBW/$, because much larger flash drives were available cheaper. Double the size of the drive using identical technology, and you double TBW ratings.

This was very clearly not true at the time for the actual implied TBW figures of even a tiny Optane drive, and is not even true today when you account for the much lower DWPD of TLC/QLC media. Do the math, $1/GB vs $0.1/GB where the actual difference in DWPD per spec sheets is more like two or three orders of magnitude, with the real-world practical one being quite possibly larger. (Have people even seen Optane actually fail in the wild due to media wear out? This happens all the time with NAND.)

> it wouldn't survive more TBW/$

Yes it would, by an almost arbitrarily large margin. You can test this out for yourself. Overwrite one of each in an endless loop. Whenever the flash based drive fails, replace it and continue. See how long it takes for the optane to fail.

You should be able to kill a typical consumer flash drive in well under a week. Even high end enterprise gear will be dead within a couple of months.

So instead of replacing every 5 years you replace every 5 years because if you need that level of performance you're replacing servers every 5 years anyway

I worked at Micron in the SSD division when Optane (originally called crosspoint “Xpoint”) was being made. In my mind, there was never a real serious push to productize it. But it’s not clear to me whether that was due to unattractive terms of the joint venture or lack of clear product fit.

There was certainly a time when it seemed they were shopping for engineers opinions of what to do with it, but I think they quickly determined it would be a much smaller market anyway from ssds and didn’t end up pushing on it too hard. I could be wrong though, it’s a big company and my corner was manufacturing and not product development.

I worked at Intel for a while and might be able to explain this.

There were/are often projects that come down from management that nobody thinks are worth pursuing. When i say nobody, it might not just be engineers but even say 1 or 2 people in management who just do a shit roll out. There are a lot of layers of Intel and if even one layer in the Intel Sandwich drag their feet it can kill an entire project. I saw it happen a few times in my time there. That one specific node that intel dropped the ball on kind of came back to 2-3 people in one specific department, as an example.

Optane was a minute before I got there, but having been excited about it at the time and somewhat following it, that's the vibe I get from Optane. It had a lot of potential but someone screwed it up and it killed the momentum.

Are you referring to the Intel 10nm struggles in your reference to 2-3 people?

This is actually insane. Do you mean 2-4 people in one department basically killed Intel? Roll to disbelief.

It's somewhat plausible that a small group of people in one department were responsible for the bad bets that made their 10nm process a failure. But it was very much a group effort for Intel to escalate that problem into the prolonged disaster. Management should have stopped believing the undeliverable promises coming out of their fab side after a year or two, and should have started much sooner to design chips targeting fab processes that actually worked.

Yes this is pretty common in large enterprise-ey tech companies that are successful. There are usually a small group of vocal members that have a strong conviction and drive to make a vision a reality. This is contrary to popular belief that large companies design by committee.

Of course it works exceptionally well when the instinct turns out to be right. But can end companies if it isn’t.

A friend was working at Micron on a rackmount network server with a lot of flash memory, I didn't ask at the time what kind of flash it used. The project was cancelled when nearly finished.

A ways back, I wrote a sort of database that was memory-mapped-file backed (a mistake, but I didn’t know that at the time), and I would have paid top dollar for even a few GB of NVDIMMs that could be put in an ordinary server and could be somewhat straightforwardly mounted as a DAX filesystem. I even tried to do some of the kernel work. But the hardware and firmware was such a mess that it was basically a lost cause. And none of the tech ever seemed to turn into an actual purchasable product. I’m a bit suspicious that Intel never found product-market fit in part because they never had a credible product on the NVDIMM side.

Somewhere I still have some actual battery-backed DIMMs (DRAM plus FPGA interposer plus awkward little supercapacitor bundle) in a drawer. They were not made by Intel, but Intel was clearly using them as a stepping stone toward the broader NVDIMM ecosystem. They worked on exactly one SuperMicro board, kind of, and not at all if you booted using UEFI. Rebooting without doing the magic handshake over SMBUS [0] first took something like 15 minutes, which was not good for those nines of availability.

[0] You can find my SMBUS host driver for exactly this purpose on the LKML archives. It was never merged, in part, because no one could ever get all the teams involved in the Xeon memory controller to reach any sort of agreement as to who owned the bus or how the OS was supposed to communicate without, say, defeating platform thermal management or causing the refresh interval to get out of sync with the DIMM temperature, thus causing corruption.

I’m suspicious that everything involved in Optane development was like this.

If you're thinking about reliability in terms of 9s you should probably not be depending on a single machine. Reboots could take hours and be fine if your architecture is set up for reliability.

IMO, the reason they didn't sell is the ideal usage for them is pairing them with some slow spinning disks. The issue Optane had is that SSD capacity grew dramatically while the price plummeted. The difference between Optane and SSDs was too small. Especially since the M.2 standard proliferated and SSDs took advantage of PCI-E performance.

I believe Optane retained a performance advantage (and I think even today it's still faster than the best SSDs) but SSDs remain good enough and fast enough while being a lot cheaper.

The ideal usage of optane was as a ZIL in ZFS.

That may have been the ideal usage back in the day, but ideal usage now is just for setting up swap. Write-heavy workloads are king with Optane, and threshing to swap is the prototypical example of something that's so write-heavy it's a terrible fit for NAND. Optane might not have been "as fast as DRAM" but it was plenty close enough to be fit for purpose.

That would be fine if I could put it in an M.2 slot. But all my computers already have RAM in their RAM slots, and even if I had a spare RAM slot, I don't know that I'd trust the software stack to treat one RAM slot as a drive...

And their whole deal was making RAM persistent anyway, which isn't exactly what I want.

Optane M.2-format hardware exists.

Interesting, all I ever saw advertised was that weird persistent kinda slow RAM stick. Does the M.2 version just show up as a normal block device or is that too trying to be persistent RAM?

Just normal (and fast) block storage.

Iirc it wasn't great because higher power == more heat though

That could be addressed with a small NVMe heatsink. They're available and their use is advised already for NAND PCIe 4.0 and 5.0 hardware, but they would fit the Optane use just as well.

Not just capacity but SSD speeds also improved to the point it was good enough for many high memory workloads.

> The ideal usage of optane was as a ZIL in ZFS.

It was also the best boot drive money could buy. Still is, I think, though other comments in the thread ask how it compares against today's best, which I'd also love to see.

This concept was very popular back in the days when computers used to boot from HDD, but now it doesn't make much sense. I wouldn't notice If my laptop boots for 5 sec instead of 10.

At the time of their introduction Optane drives were noticeably faster to boot your machine than even the fastest available Flash SSD. So in a workstation with multiple hard drives installed anyway, buying one to boot off of made decent sense.

If they had been cheaper, I think they'd have been really, really popular.

What concept was very popular in those days?

By my reckoning, there was zero overlap between the period of time where a reasonable computer configurer would pick a hard drive to boot from and the period of time where Optane was available.

And even for the general concept of a cache drive, I don't think I've ever seen it do well in the mainstream. Just a few niche roles, and some hybrid drives that sucked because they had small flash chips and only used them as a read cache, not a write cache.

Intel did a spectacularly poor job with the ecosystem around the memory cells. They made two plays, and both were flops.

1. “Optane” in DIMM form factor. This targeted (I think) two markets. First, use as slower but cheaper and higher density volatile RAM. There was actual demand — various caching workloads, for example, wanted hundreds of GB or even multiple TB in one server, and Optane was a route to get there. But the machines and DIMMs never really became available. Then there was the idea of using Optane DIMMs as persistent storage. This was always tricky because the DDR interface wasn’t meant for this, and Intel also seems to have a lot of legacy tech in the way (their caching system and memory controller) and, for whatever reason, they seem to be barely capable of improving their own technology. They had multiple serious false starts in the space (a power-supply-early-warning scheme using NMI or MCE to idle the system, a horrible platform-specific register to poke to ask the memory controller to kindly flush itself, and the stillborn PCOMMIT instruction).

2. Very nice NVMe devices. I think this was more of a failure of marketing. If they had marketed a line of SSDs that, coupled with an appropriate filesystem, could give 99% fsync latency of 5 microseconds and they had marketed this, I bet people would have paid. But they did nothing of the sort — instead they just threw around the term “Optane” inconsistently.

These days one could build a PCM-backed CXL-connected memory mapped drive, and the performance might be awesome. Heck, I bet it wouldn’t be too hard to get a GPU to stream weights directly off such a device at NVLink-like speeds. Maybe Intel should try it.

One of the many problems was trying to limit the use of Optane to Intel devices. They should have manufactured and sold Optane memory and let other players build on top of it at a low level.

> Optane memory

Which “Optane memory”? The NVMe product always worked on non-Intel. The NVDIMM products that I played with only ever worked on a very small set of rather specialized Intel platforms. I bet AMD could have supported them about as easily as Intel, and Intel barely ever managed to support them.

The consumer "Optane memory" products were a combination of NVMe and Intel's proprietary caching software, the latter of which was locked to Intel's platforms. They also did two generations of hybrid Optane+QLC drives that only worked on certain Intel platforms, because they ran a PCIe x2+x2 pair of links over a slot normally used for a single X2 or x4 link.

Yes, the pure-Optane consumer "Optane memory" products were at a hardware level just small, fast NVMe drives that could be use anywhere, but they were never marketed that way.

I feel like this is proving my point. You can’t read “Optane” and have any real idea of what you’re buying.

Also… were those weird hybrid SSDs even implemented by actual hardware, or were they part of the giant series of massive kludges in the “Rapid Storage” family where some secret sauce in the PCIe host lied to the OS about what was actually connected so an Intel driver could replace the OS’s native storage driver (NVMe, AHCI, or perhaps something worse depending on generation) to implement all the actual logic in software?

It didn’t help Intel that some major storage companies started selling very, very nice flash SSDs in the mean time.

> were those weird hybrid SSDs even implemented by actual hardware, or were they part of the giant series of massive kludges

They were definitely part of the series of massive kludges. But aside from the Intel platforms they were marketed for, I never found a PCIe host that could see both of the NVMe devices on the drive. Some hosts would bring up the x2 link to the Optane half of the drive, some hosts would bring up the x2 link to the QLC half of the drive, but I couldn't find any way to get both links active even when the drive was connected downstream of a PCIe switch that definitely had hardware support for bifurcation down to x2 links. I suspect that with appropriate firmware hacking on the host side, it may have been possible to get those drives fully operational on a non-Intel host.

Why on Earth did Intel implement this as a 2x2 device? They could have implemented multiple functions or they could have used a PCIe switch or they could have exposed their device as an NVMe device with multiple namespaces, etc. (I won’t swear that all of these would have worked nicely. But all of them would have performed better than arbitrarily splitting the link in half.)

Maybe they didn’t own any of the IP for the conventional SSD part and couldn’t make it play ball?

Or perhaps they just made a number of incredibly poor decisions. They seem to have been doing that for the better part of a couple decades now.

The Optane side of the drive used the same x2 controller as the pure-Optane cache drives. The NAND side used a Silicon Motion controller, same as their consumer QLC drives of the era. They almost literally just crammed their two existing consumer products onto one PCB and shipped it. Intel was never interested enough in the consumer applications of Optane to design a good, useful SSD controller around it, and they weren't going to let a third-party like Silicon Motion make an Optane-compatible controller.

Exactly. I happen to have all AMD sitting around here, and buying my first Optane devices was a gamble, because I had no idea if they'd work. Only reason I ever did, is they got cheap at one point and I could afford the gamble.

That uncertainty couldn't have done the market any favors.

This.

Optane died because Intel wanted to kill AMD and ARM with it. Killed Intel and a great technology along with it.

In "databases and journals" you rarely update just one byte, you do a transaction that updates data, several indexes and metadata. All of that needs to be atomic.

Power failure can happen in between any of "1 byte updates with crazy latencies." However small latency is, power failure is still faster. Usually, there is a write ahead or some other log that alleviates the problem, this log is usually written in streaming fashion.

What is good, though, is that "blast radius" [1] of failure is smaller than usual - failed one byte write rarely corrupts more that one byte or cache line. SQLite has to deal with 512 (and even more) bytes long possible corruptions on most disks, with Optane it is not necessarily so. So, less data to copy, scan, etc.

[1] https://sqlite.org/psow.html

It's not. You won't be writing one byte, ever (even if you had layers that actually supported less-than-block writes), because the overhead of instruction would be massive and you'd be murdering both latency and bandwidth for anything non-trivial

> However small latency is, power failure is still faster.

A fancy switching power supply with a friendly power factor (looks like a resistive load, rather than drawing more amps during the lower voltage parts of the waveform) actually will have non-zero fall time when suddenly unplugged.

I never understood what they're meant to do. Intel seemed to picture some future where RAM is persistent; but they were never close to fast enough to replace RAM, and the option to reboot in order to fix some weird state your system has gotten itself into is a feature of computers, not a problem to work around.

When the PDIMMs were used with an appropriate file system + kernel, it was pretty cool. NTFS + DAX + kernel support yielded a file system where mmap’ing didn’t page fault. No page faults because the file content is already there, instantly.

So if you had mmap heavy read/write workloads… you could do some pretty cool stuff.

For a good technical explanation at the physical level of a memory cell:

https://pcper.com/2017/06/how-3d-xpoint-phase-change-memory-...

Related: "High-bandwidth flash progress and future" (15 comments), https://news.ycombinator.com/item?id=46700384

In an era of RAM shortages and quarterly price increases, Optane remains viable for swap and CPU/GPU cache.

Yeah, I've wondered if we might see a revival of this kind of technology.

in an era of shortages, if there was an optane factory today ready to print money...

Secondary market surplus pricing (~$1/GB) value accrues to the buyer..

> (~$1/GB)

Isn't that actually crazy good, even insane value for the performance and DWPD you get with Optane, especially with DRAM being ~$15/GB or so? I don't think ~$1/GB NAND is anywhere that good on durability, even if the raw performance is quite possibly higher.

I’ve been considering buying 8x64g models and setting them as equal priority swap disks (to mitigate the low throughput) for this exact reason.

Can confirm doing so is awesome. Get some slightly bigger ones and partition them for additional use as zil. They're extremely satisfying to use, and depressing to remember that we'll never see their like again.

Do you have any more details? This is such a niche idea, that I’d be buying blind.

Sure! This is more or less how I'm using Optane in my storage box:

Two of U.2x4 to PCIe x16 riser cards, one loaded with 960GB Intel-branded Optanes, one with 1.5TB IBM-branded. PCIe bifurcation is set up in the BIOS to let them all come up properly, where they just show as regular NVMe. Riser cards like this can easily be substituted for PCIe to SAS/Oculink to U.2 cables, if that would be more accommodating to your chassis.

Once they all come up, partition them for your preferred split of swap and ZFS special. Swap should have them all mounted with the same priority and discard=pages, I also recommend setting up zswap (not zram swap) with lz4 as an additional layer of fast, evictable memory pool, as well as `vm.overcommit_memory=2` and `vm.swappiness=150`. This will effectively give you really good memory tiering for workloads and file cache.

When adding the other partitions to ZFS, use `-ashift=12 special mirror dev dev special mirror dev dev ...`. ZFS special covers all metadata, the intent log (sorta write cache), and optionally small files. I like to set it up so <= 8k small files get sent there, but you can probably go higher depending on how much capacity you allocate. My ~24T of allocated data ended up being ~150GB special with 8k small file, and that's with the whole pool configured with deduplication and blake3 for all hashes. Blake3 is fast as heck, but has very long hashes, so from a metadata standpoint, I'm using the most expensive option. I mitigate that a bit my setting metadata redundancy to `some`, since my metadata is effectively RAID10 anyway.

With some extra NVMe/Optane allocated to regular ZFS read cache, and all my spinning-rust data VDEVs also as RAID10, it's almost like having the whole array in memory, or at least on fast flash. Eliminating metadata from your drives seeking and letting them be written nearly instantly with Optane does wonderful things for spinning rust :)

These are absolute beasts for database servers, and definitely needs to make a comeback.

They suck for large sequential file access, but incredible for small random access: databases.

My understanding is Optane is still unbeaten when it comes to latency. Has anyone examined its use as an OS volume, compared to today's leading SSD's? I know the throughput won't be as high, but in my experience that's not as important to how responsive your machine feels as latency.

Before people claim it doesn't matter due to OS write buffering, I should point out a) today's bloated software and the many-layered, abstracted I/O stack it's built on tends to issue lots of unnecessary flushes, b) read latency is just as important as write (if not moreso) to how responsive your OS feels, particularly if the whole thing doesn't fit in (or preload to) memory.

I configured a hetzner ax101 bare metal server with a 480GB 3d xpoint ssd some years ago. It’s used as the boot volume and it seems fast despite the server being heavily over provisioned, but I can’t really compare because I don’t have a baseline without.

> Has anyone examined its use as an OS volume, compared to today's leading SSD's?

Late last year I switched from a 1.5tb Optane 905P to a 4tb WD Blue SN5000 NVMe drive in a gaming machine and saw improved load times, which makes sense given the read and write speeds are ~double. No observable difference otherwise.

I'm sure that's not the use case you were looking for. I could probably tease out the difference in latency with benchmarks but that's not how I use the computer.

The 905P is now in service as an SSD cache for a large media server and that came with a big performance boost but the baseline I'm comparing to is just spinning drives.

Thanks, that's helpful real-world feedback (not that I wouldn't also be interested in some synthetic benchmark comparisons from someone else).

We benchmarked three of the popular Optane NVMe SSDs about three years ago. There was a short window when they were on clearance and a popular choice as a cache SSD in TrueNAS.

https://pcpartpicker.com/forums/topic/425127-benchmarking-op...

You can compare their benchmarks with the other almost 400 SSDs we've benchmarked. Most impressive is that three years later they are still the top random read QD1 performers, with no traditional flash SSD coming anywhere close:

https://pcpartpicker.com/products/internal-hard-drive/benchm...

They are amazing for how consistent and boring their performance is. Bit level access means no need for TRIM or garbage collection, performance doesn't degrade over time, latency is great, and random IO is not problematic.

Unfortunately a gaming machine workload is so read-heavy that I wouldn't expect Optane to square up well. Gaming is all about read speed and overall capacity. You need that heavy I/O mix, especially with low latency deadlines, to see gains from Optane. That limited target use case, coupled with ignorant benchmarking, always limited them.

I have a 16 GiB Optane NVMe M.2 drive in my router as a boot drive, running OpenWRT.

It's so incredibly fast and responsive that the LuCI interface completely loads the moment I hit enter on the login form.

I run two 1.5TB Optanes in raid-0 with XFS (I picked them up for $300 each on sale about two years ago). These are limited to PCIE 3.0 x4 (about 4GB/s max each). I also have a 64GB optane drive I use as my boot drive.

It's hard to tell you, because it's subjective, I don't swap back and forth between an SSD and the optane drives. I have my old system, which has a 2TB Samsung 980 Pro NVME drive (PCIE 4.0 x4, or 8GB/s max) as root, and a Sabrent rocket 4 plus 4TB drive secondary (also PCIE 4.0), so I ran sysbench on both systems, so I could share the differences. (Old system 5950X, new system 9950X3D).

It feels snappier, especially when doing compilations...

Sequential reads: I started with a 150GB fileset, but it was being served by the kernel cache on my newer system (256GB RAM vs 128GB on the old), so I switched to use 300GB of data, and the optanes gave me 5000 MiB/s for sequential read as opposed to 2800 MiB/s for the 980 Pro, and 4340 MiB/s for the Rocket 4 Plus.

Random writes alone (no read workload) The optane system gets 2184 MiB/s, the 980 Pro gets 32 MiB/s, and the Rocket 4 Plus gets 53 MiB/s.

Mixed workload (random read/write) The optanes get 725/483 as opposed to 9/6 for the 980 Pro, and 42/28 for the Rocket 4 Plus.

2x1.5TB Optane Raid0: Prep time: `sysbench fileio --file-total-size=150G prepare` 161061273600 bytes written in 50.41 seconds (3047.27 MiB/sec).

    Benchmark:
    `sysbench fileio --file-total-size=150G --file-test-mode=rndrw --max-time=60 --max-requests=0 run`
    WARNING: --max-time is deprecated, use --time instead
    sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)

    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from current time

    Extra file open flags: (none)
    128 files, 1.1719GiB each
    150GiB total file size
    Block size 16KiB
    Number of IO requests: 0
    Read/Write ratio for combined random IO test: 1.50
    Periodic FSYNC enabled, calling fsync() each 100 requests.
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random r/w test
    Initializing worker threads...

    Threads started!

    File operations:
        reads/s:                      46421.95
        writes/s:                     30947.96
        fsyncs/s:                     99034.84

    Throughput:
        read, MiB/s:                  725.34
        written, MiB/s:               483.56

    General statistics:
        total time:                          60.0005s
        total number of events:              10584397

    Latency (ms):
             min:                                    0.00
             avg:                                    0.01
             max:                                    1.32
             95th percentile:                        0.03
             sum:                                58687.09

    Threads fairness:
        events (avg/stddev):           10584397.0000/0.00
        execution time (avg/stddev):   58.6871/0.00

2TB Nand Samsung 980 Pro: Prep time: `sysbench fileio --file-total-size=150G prepare` 161061273600 bytes written in 87.15 seconds (1762.53 MiB/sec).

    Benchmark:
    `sysbench fileio --file-total-size=150G --file-test-mode=rndrw --max-time=60 --max-requests=0 run`
    WARNING: --max-time is deprecated, use --time instead
    sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)

    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from current time

    Extra file open flags: (none)
    128 files, 1.1719GiB each
    150GiB total file size
    Block size 16KiB
    Number of IO requests: 0
    Read/Write ratio for combined random IO test: 1.50
    Periodic FSYNC enabled, calling fsync() each 100 requests.
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random r/w test
    Initializing worker threads...

    Threads started!

    File operations:
        reads/s:                      594.34
        writes/s:                     396.23
        fsyncs/s:                     1268.87

    Throughput:
        read, MiB/s:                  9.29
        written, MiB/s:               6.19

    General statistics:
        total time:                          60.0662s
        total number of events:              135589

    Latency (ms):
             min:                                    0.00
             avg:                                    0.44
             max:                                   15.35
             95th percentile:                        1.73
             sum:                                59972.76

    Threads fairness:
        events (avg/stddev):           135589.0000/0.00
        execution time (avg/stddev):   59.9728/0.00

4TB Sabrent Rocket 4 Plus: Prep time: `sysbench fileio --file-total-size=300G prepare` 322122547200 bytes written in 152.39 seconds (2015.92 MiB/sec).

    Benchmark:
    `sysbench fileio --file-total-size=300G --file-test-mode=rndrw --max-time=60 --max-requests=0 run`
    WARNING: --max-time is deprecated, use --time instead
    sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)

    Running the test with following options:
    Number of threads: 1
    Initializing random number generator from current time

    Extra file open flags: (none)
    128 files, 2.3438GiB each
    300GiB total file size
    Block size 16KiB
    Number of IO requests: 0
    Read/Write ratio for combined random IO test: 1.50
    Periodic FSYNC enabled, calling fsync() each 100 requests.
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random r/w test
    Initializing worker threads...

    Threads started!

    File operations:
        reads/s:                      2690.28
        writes/s:                     1793.52
        fsyncs/s:                     5740.92

    Throughput:
        read, MiB/s:                  42.04
        written, MiB/s:               28.02

    General statistics:
        total time:                          60.0155s
        total number of events:              613520

    Latency (ms):
             min:                                    0.00
             avg:                                    0.10
             max:                                    8.22
             95th percentile:                        0.32
             sum:                                59887.69

    Threads fairness:
        events (avg/stddev):           613520.0000/0.00
        execution time (avg/stddev):   59.8877/0.00

Optane was crazy good tech, it way just too expensive at the time for mass adoption, but the benefits were so good.

Looking at those charts, besides the DWPD it feels like normal NVMe has mostly caught up. I occassionally wonder where a gen 7/8(?) optane would be today if it caught on, it'd probably be nuts.

> besides the DWPD it feels like normal NVMe has mostly caught up.

So what you mean is that on the most important metric of them all for many workloads, Flash-based NVMe has not caught up at all. When you run a write heavy workload on storage with a limited DWPD (including heavy swapping from RAM) higher performance actually hurts your durability.

The actual strength of Optane was on mixed workloads. It's hard to write a flash cell (read-erase-write cycle, higher program voltage, settling time, et cetera). Optane didn't have any of that baggage.

This showed up as amazing numbers on a 50%-read, 50%-write mix. Which, guess what, a lot of real workloads have, but benchmarks don't often cover well. This is why it's a great OS boot drive: there's so much cruddy logging going on (writes) at the same time as reads to actually load the OS. So Optane was king there.

It’s the best OS drive especially p5800x.

I don't have the inside scoop on Intel's current mess, but they definitely have a habit of killing off their coolest projects.

Realsense too

One potential application I briefly had hope for was really good power loss protection in front of a conventional Flash SSD. You only need a little compared to the overall SSD capacity to be able to correctly report the write was persisted, and it's always running, so there's less of a 'will PLP work when we really need it?' question. (Maybe there's some use as a read cache too? Host RAM's probably better for that, though.) It's going to be rewritten lots of times, but it's supposed to be ready for that.

It seems like there's a very small window, commercially, for new persistent memories. Flash throughput scales really cost-efficiently, and a lot is already built around dealing with the tens-of-microseconds latencies (or worse--networked block storage!). Read latencies you can cache your way out of, and writers can either accept commit latency or play it a little fast and loose (count a replicated write as safe enough or...just not be safe). You have to improve on Flash by enough to make it worth the leap while remaining cheaper than other approaches to the same problem, and you have to be confident enough in pulling it off to invest a ton up front. Not easy!

> It seems like there's a very small window, commercially, for new persistent memories. Flash throughput scales really cost-efficiently

Flash is no bueno for write-heavy workloads, and the random-access R/W performance is meh compared to Optane. MLC and SLC have better durability and performance, but still very mid.

Any decent SSD has capacitor (enterprise) or battery backed (phones) DRAM. Therefore, a sync write is just “copy the data to an I/O buffer over PCIe”.

For databases, where you do lots of small scattered writes, and lots of small overwrites to the tail of the log, modern SSDs coalesce writes in that buffer, greatly reducing write wear, and allowing the effective write bandwidth to exceed the media write bandwidth.

These schemes are much less expensive than optane.

I have tried multiple enterprise SSD's, for sync writes. Nothing comes close to Optane Dimm, even Optane NVMe is 10x slower than PDIMMS.

https://forums.servethehome.com/index.php?threads/so-i-teste...

Yeah, but what percentage of the time was the CPU stuck blocked on the memory bus for the two workloads?

Also, how was it guaranteeing the page writes were atomic?

You are correct that writes are not guaranteed to be atomic. PostgreSQL has a feature to handle torn pages: full_page_writes = on, which is enabled by default. This means you do double write.

For RocksDB I believe torn pages are not a problem as SST files are immutable. But correct me if I am wrong.

> One potential application I briefly had hope for was really good power loss protection in front of a conventional Flash SSD.

That was never going to work out. Adding an entirely new kind of memory to your storage stack was never going to be easier or cheaper than adding a few large capacitors to the drive so it could save the contents of the DRAM that the SSD still needed whether or not there was Optane in the picture.

I feel sorry about the situation. From my perspective Optane was a godsend for databases. I was contemplating building a system. Could've been a pinnacle of vertical scalability for cheap.

Around the time of Optane's discontinuation, the rumor mill was saying that the real reason it got the axe was that it couldn't be shrunk any, so its costs would never go down. Does anyone know if that's true? I never heard anything solid, but it made a lot of sense given what we know about Optane's fab process.

And if no shrink was possible, is that because it was (a) possible but too hard; (b) known blocks to a die shrink; or (c) execs didn't want to pay to find out?

That's at least physically half-plausible, but it would be a terrible reason if true. 3.5 in. format hard drives can't be shrunk any, and their costs are correspondingly high, but they still sell - newer versions of NVMe even provide support for them. Same for LTO tape cartridges. Perhaps they expected other persistent-memory technologies to ultimately do better, but we haven't really seen this.

Worth noting though that Optane is also power-hungry for writes compared to NAND. Even when it was current, people noticed this. It's a blocker for many otherwise-plausible use cases, especially re: modern large-scale AI where power is a key consideration.

> 3.5 in. format hard drives can't be shrunk any,

You're looking at the entirely wrong kind of shrinking. Hard drives are still (gradually) improving storage density: the physical size of a byte on a platter does go down over time.

Optane's memory cells had little or no room for shrinking, and Optane lacked 3D NAND's ability to add more layers with only a small cost increase.

I think it was killed primarily because the DIMM version had a terrible programming API. There was no way to pin a cache line, update it and flush, so no existing database buffer pool algorithms were compatible with it. Some academic work tried to address this, but I don’t know of any products.

The SSD form factor wasn’t any faster at writes than NAND + capacitor-backed power loss protection. The read path was faster, but only in time to first byte. NAND had comparable / better throughput. I forget where the cutoff was, but I think it was less than 4-16KB, which are typical database read sizes.

So, the DIMMs were unprogrammable, and the SSDs had a “sometimes faster, but it depends” performance story.

It sounds like they didn't do a good job of putting the DIMM version in the hands of folks who'd write the drivers just for fun.

The read path is sort of a wash, but writes are still unequalled. NAND writes feel like you're mailing a letter to the floating gate...

Isn't this addressed by newer PCIe standards? Of course, even the "new" Optane media reviewed in OP is stuck on PCIe 4.0...

No; the issue with the DIMMs wasn’t drivers. The issue was that the only people allowed to target the DIMMs directly were the xeon hardware team.

There was a startup doing good work with similar storage chips that were pin (BGA) compatible with standard memory. Not sure what happened to them. That’d be easier to program than xpoint.

As for the new PCIe standard (you probably mean CXL), that’s also basically dead on arrival. The CPU is the power and money bottleneck for the applications it targets, so they provide a synchronous hardware API that stalls the processor pipeline when accessing high-latency devices.

Contrast this to NVMe, which can be set up to either never block the CPU or amortize multiple I/Os per cache miss.

Companies like NVIDIA are already able to maintain massive I/O concurrency over PCIe without CXL, because they have a programming model (the GPU) that supports it. CXL might be a small win for that.

Interesting perspective re CXL synchronous API. Wouldn't things like OOO execution and speculation help with that? And anyway the latency is supposed to be comparable to NUMA latency, is that really such a deal breaker?

The DIMMs were their own shitshow and I don't know how they even made it as far as they did.

The SSDs were never going to be dominant at straight read or write workloads, but they were absolutely king of the hill at mixed workloads because, as you note, time to first byte was so low that they switched between read and write faster than anything short of DRAM. This was really, really useful for a lot of workloads, but benchmarkers rarely bothered to look at this corner... despite it being, say, the exact workload of an OS boot drive.

For years there was nothing that could touch them in that corner (OS drive, swap drive, etc) and to this day it's unclear if the best modern drives still can or can't compete.

Flash has the same shrink problem. And the solution for Optane was the same: go 3D

I don't think the shrink problem is at all the same for the two technologies. There are some really weird materials and production steps in Optane that are simply not present when making Flash cells.

durability drops quickly with shrinking flash, we won't see much smaller cells, the growth has been MLC-TLC-> QLC and stacking

I think a factor in Optane's demise was dishonesty on the part of low-end laptop manufacturers.

You were a poor student looking for an entry-level cheap laptop and saw a lot of models with 4GB [RAM] "memory" and suddenly there was this one model with "20 GB memory" for the same price. Seemed attractive to the regular guy, but this in fact was 4GB RAM + 16 GB Optane non-volatile storage (and maybe a paltry 32 GB SSD for the rest). Optane would be treated as a drive for storing the Windows OS.

That conflation of Optane storage as "memory", hinting that this was equal to RAM, turned many people against it once they fell victims to that bait-and-switch.

I never bought one because I only saw Optane in these low spec configurations. I didn't see any use for a tiny SSD. I only learned more about it after it was discontinued.

Is there any alternative today? I benched my brand new pcie5 ssd today and while it does over 10GB/s sequential data for random small chunks it barely exceeds 70MB/s. Meanwhile Optane was 5 times faster than that.

Maybe we can also mention the HP Memristor here.

Oh I was so excited for that. I devoured any news or blogs or rumours about that immediately!

I wish it had made it to thumb drives for its superior data retention. It doesn't work by holding a charge; it's phase-change, so it retains data longer.

I encountered FRAM chips in one PCB I worked with. It's like flash, but holds data forever and does not wear. One could easily create usb drive of small capacity using these chips. They are a bit expensive, but that longevity could be a market differentiator. So tech is there if you think you can sell it.

Did it ever have real input costs which justified its pricing or was it totally a synthetic price to cover an imagined value above base, and some IPR?

I may be wrong, but I tend to think the fab cost has very little to do with the complexity, but not nothing, and the outcome cost is very unrelated to price at a viable yield.

If optane failed, it was probably because Intel didn't like the economics of pricing it to succeed.

I still think it has a lot of potential, and is the logical next storage system, because of its performance advantages over NAND flash. Because input costs can come down.

"here's one I prepared earlier" would be cute. I think other people's threads pointing out to scale globally it has to be something anyone can do, and right now Intel appear to have set the RAND conditions or whatever IPR they lock into this higher than other people want to pay.

Surprised Optane wasn't kept on life support by hyperscalers and kicked in to overdrive as the demand for AI memory spiked.

The fab was already taken apart and sold by then.

Sure, they were expensive but they have great endurance and sustained read and write speeds. I use one in my car for camera recordings. I had gone through several other drives but this one has been going on 3 or 4 years now without issue. I have a couple more in use too. It's a shame this tech is going away because it's excellent.

Did anyone ever see retention issues like this guy reported on one of his older models?

https://goughlui.com/2024/07/28/tech-flashback-intel-optane-...

That's data retention issues on the very first read-through of the media after sitting in cold storage for many years, with subsequent performance returning to normal. It's definitely something to be aware of (and kudos to the blog poster for running that experiment) but worn-out NAND will behave a lot worse than that.

Retention issues with completely new Optane is not great compared to SLC NAND, which Optane was supposed to beat. I have USB drives with the latter, probably a few dozen cycles at most, yet they have had no problems keeping data intact for over a decade.

That's not completely new, it's just new old stock. That's the whole point of why the blogger ran that experiment.

Perhaps "unworn" would be a better way to phrase it, as it had zero program/erase cycles. Even unworn NAND flash has better retention than that.

I’m still sad they discontinued them. What’s the alternative now does anything come close?

Small sizes are on secondary market for ~$1/GB.

Which is a bargain compared to what DRAM costs today. If you just include the bare minimum of DRAM for a successful boot and immediately set up the entire "small" Optane drive as swap, that's a viable workstation-class system for comparative peanuts. You can't do this with NAND because the write workload of swap kills the media (I suppose it becomes viable if you monitor SMART wearout indicators and heavily overprovision the storage to leverage the drive's pSLC mode, but you're still treating $~0.10/GB hardware as a consumable and that will cost you) and of course you can't do it with spinning rust because the media is too slow.

All those nice numbers are just beaten by the unit cost. And the ecosystem is a mess.

My kingdom for a MicroSD card with Optane inside. My dashcam wants it soooo badly.

Its not Optane but there are SLC microSD cards!

I have a few of them, and yes, they're worth every penny!

(For posterity: Mouser and Digikey et alia carry these things and let you search/filter by cell type. They're not cheap.)

Fabs are expensive and all, but maybe running a right-sized fab could have still been profitable at making optane for low-latency work that it was so good at. Even moreso with RAM prices as they are.

PE popped already, now Private credit is already popping, having been in too many bubbles, Datacenter bubble popped them. The whole system just isn't talking about it because the media isn't doing their job to report. There are no more good loans to make, the ROI is nil and the interest rate is spot plus 5.5%, and businesses don't want to pay 10% interest on a loan. We're ina zombie cycle

the tech had so much potential, but it was during the era when Intel dominated. other than the pcie add-in cards that can act has fast cache, all the major use cases were down to their whims. platform support (or the lack thereof) killed it before it could be realized.

too bad there was no apple moment (e.g. with thunderbolt) where a third-party was allowed to make it more widespread.

I've got a bunch of them in my homelab. They're good if applied to tasks that leverage their attributes.

I use two Intel 905 SSDs as mirrored cache devices for ZFS.

Why would you mirror a cache device?

I think a read cache device gets set up like RAID0, interleaved reads rather than redundant data.

Examples of auxiliary devices where you want redundancy: there can be a write cache (the ZFS Intent Log, or ZIL). You can also dedicate a fast device for hot items like the deduplication tables, or a dedicated device for tiny files where data can be stored directly in the directory data, rather than allocating a separate data block.

Now do Intel's HBM/CPU Max

Reply on news.ycombinator.com