The combination of NVME's IOPS and throughput, particularly for sequential reads/writes like you'd have with something like huge pages, begins making it a reasonable consideration for caching/secondary memory in certain situations.
edit: s/NVME/non-volatile/
With 24X NVMe servers now common and cheap (AMD EPYC) you can get 96GB/s (theoretical max, and real is not far off) onto NVMe which is same order of magnitude as memory bandwidth.
When PCI-e 4 comes out next year the bandwidth will double and SSD manufacturers will have no problem doubling the performance of NVMe devices once the interface supports it so at that point the bandwidth on NVMe will surpass RAM.
The major caveat is that sequential read/write is much faster than random (this can be offset to a certain extent if you can queue up many random requests at the same time but each random request has a very high latency) so I would expect that switching from 4K to 2MB pages would have significant advantages on the storage side beyond efficiencies gained in the kernel.
So the non-volatility is indirectly related to why those memories are more cost-effective than DRAM, but they still have the array structure that allows for much lower access times than a hard drive.
This has been in SSDs for long enough without being hailed as a game changer, which is why I question anything common between the two that appears to be called out as the cause.
> Intel's 3D XPoint have fundamental advantages over DRAM that guarantee they'll be cheaper
This may be true for the producer, but in a monopoly situation, one could reasonably expect any savings will be collected as profit rather than passed on to the consumer.
It's only been in the past few years that flash memory has been accessed through interfaces that weren't made for hard drives. NVMe is recent enough that it's no surprise to see that the conservative, slow-evolving OS virtual memory subsystem still hasn't been entirely re-architected to account for the new possibilities that fast, low-latency storage brings.
Just how few years are we talking about? From what I recall, Fusion-IO's model didn't use a hard disk interface, and their products were in the sub-millisecond latency range over 6 years ago, maybe even a full decade.
At the time (and more or less continuously since then), I've questioned why anyone would want to pay such a huge premium over disk-imitating flash (that is, enough of it to match the storage, throughput, and IOPS).
Low latency would be a credible explanation, although I don't recall anything in the way of even benchmarks showing the difference with real-world software like a database.
All that said, unless I'm missing something like an order of magnitude drop in latency, I'm wondering why.. this time it's different?