The final step for huge-page swapping
lwn.net
lwn.net
It's not clear to me why the nonvolatile part makes any difference to swap. I'd expect it would make more sense to attach a volatile, RAM-based SSD [1], maybe via PCIe or even once removed via storage fabric.
Perhaps it's just incidental, and it's merely price and/or capacity that's important, considering that it's attached via the memory bus.
[1] Not that I've ever seen such a product with a high enough capacity for a low enough price. It seems like it could be a way to recycle old and/or slow RAM, maybe a startup idea, but I'm no hardware guy.
One solution is to stream the mip levels, and to create virtualized mipmaps. This basically lets you create a 128k by 128k texture, which works fine because only small blocks are loaded. It’s literally the same thing as virtual memory, but for texture memory.
One downside is that when you shut down the game / simulation, all of that data no longer exists. It was virtual. It either streamed from the sever or you packed it to disk somewhere. Either way you have to set it up again the next time the program starts.
I don’t know whether the present topic would let you bypass this limitation, but if there was some way to start a program with a huge amount of memory allocations preloaded, that would be very attractive. And if you can reboot without losing that, then suddenly you can start crafting persistent worlds that load instantly with ~infinite resolution.
This is the dream that voxel tech was supposed to make into reality, and it will only become more of a concern over time as mixed reality tech matures.
[1] https://msdn.microsoft.com/en-us/library/windows/desktop/hh7...
[1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
-ss
There's no single API on Windows that does the myriad of things that madvise does.
My suspicion is that it’s mainly because memory mapping files, while safe for reading, have a whole bunch of edge cases where the file can accidentally be corrupted.
I can't imagine that it would, since it's just swapping. That's why I asked the question.
How to take advantage of the nonvolatile aspect of that memory as you describe is a much broader, more complicated topic, involving bootloaders and kernels.
The combination of NVME's IOPS and throughput, particularly for sequential reads/writes like you'd have with something like huge pages, begins making it a reasonable consideration for caching/secondary memory in certain situations.
edit: s/NVME/non-volatile/
So the non-volatility is indirectly related to why those memories are more cost-effective than DRAM, but they still have the array structure that allows for much lower access times than a hard drive.
This has been in SSDs for long enough without being hailed as a game changer, which is why I question anything common between the two that appears to be called out as the cause.
> Intel's 3D XPoint have fundamental advantages over DRAM that guarantee they'll be cheaper
This may be true for the producer, but in a monopoly situation, one could reasonably expect any savings will be collected as profit rather than passed on to the consumer.
It's only been in the past few years that flash memory has been accessed through interfaces that weren't made for hard drives. NVMe is recent enough that it's no surprise to see that the conservative, slow-evolving OS virtual memory subsystem still hasn't been entirely re-architected to account for the new possibilities that fast, low-latency storage brings.
Just how few years are we talking about? From what I recall, Fusion-IO's model didn't use a hard disk interface, and their products were in the sub-millisecond latency range over 6 years ago, maybe even a full decade.
At the time (and more or less continuously since then), I've questioned why anyone would want to pay such a huge premium over disk-imitating flash (that is, enough of it to match the storage, throughput, and IOPS).
Low latency would be a credible explanation, although I don't recall anything in the way of even benchmarks showing the difference with real-world software like a database.
All that said, unless I'm missing something like an order of magnitude drop in latency, I'm wondering why.. this time it's different?
With 24X NVMe servers now common and cheap (AMD EPYC) you can get 96GB/s (theoretical max, and real is not far off) onto NVMe which is same order of magnitude as memory bandwidth.
When PCI-e 4 comes out next year the bandwidth will double and SSD manufacturers will have no problem doubling the performance of NVMe devices once the interface supports it so at that point the bandwidth on NVMe will surpass RAM.
The major caveat is that sequential read/write is much faster than random (this can be offset to a certain extent if you can queue up many random requests at the same time but each random request has a very high latency) so I would expect that switching from 4K to 2MB pages would have significant advantages on the storage side beyond efficiencies gained in the kernel.
Why not just attach that RAM as system memory in such a situation? The point of using nonvolatile is that it's cheaper and consumes less energy
Because the number of available DIMM slots, and therefore total capacity, is scarce. Increases in capacity require increases in density, which carry a cost (and presumably power density and/or speed) premium.
System memory scarcity is why swap is a thing in the first place.
Having that extra RAM on a slower, farther bus, as swap, means it doesn't have quite as stringent performance requirements (or abilities) with the benefit of increased overall capacity. This tradeoff sounds familiar.
> The point of using nonvolatile is that it's cheaper
That has yet to be seen. It stands to reason that it'll be cheaper than the RAM that would otherwise go in the same DIMM slot, but it won't perform as well, either. We can be confident it will perform much better than even PCIe/NVMe SSDs.
Is it cheaper than a generation or two ago's RAM in a big array (which, again, I know doesn't actually exist)?
The main question is, given Intel's monopoly, what the real world price per performance is going to be like.
> consumes less energy
I hadn't really thought about this as a significant consideration, although at very large memory capacities a few watts here and there do add up, and it's certainly important when comparing anything new against anything old. Still, it seems like it would just be another cost factor.
That said, why would size (density) have to do with being economical? Are the required memory controllers to expensive, because there simply aren't any other than inside CPUs? The train of logic needs explaining.
It is the same reason why I don't reuse old hds, it does make economic sense. I would rather overpay for a 12 tb drive than get 12 1tb drives for free. The trouble, increased failure rates, the space usage etc is just not worth it.
A few ssds in a high speed non redundant raid would be much better.
Or a new class of volitile slow ram that has tons of space would be interesting. So more space at lower cost than ram but slower. That would be interesting. Give me a few 128GB sticks please.
You've said this already, but both of these statements need unpacking and quantifying, especially "a ton of space".
There's been a persistent myth that, in datacenters, it's space that's the expensive and/or scarce resource. It isn't. It's power and cooling. Only in the last 5-6 years have we seen the construction of datacenter facilities that could handle half-U densities, let alone what some "blade" systems were trying to do.
> It is the same reason why I don't reuse old hds, it does make economic sense. I would rather overpay for a 12 tb drive than get 12 1tb drives for free. The trouble, increased failure rates,
This is a false equivalence, since RAM doesn't have moving parts. A better comparison would be if you discovered a box of brand new drives, still sealed in their bags with dessicant.
Even ignoring the fact that 12 drives will have much better performance than a single one, would you still just blindly throw them away to buy the single one, without considering if it would be cheaper to buy 12 disk slots instead?
If the answer is "yes", this is inconsistent with "economic".
> the space usage etc is just not worth it.
I've already addressed the space usage as a red herring. What's the "etc"?
> A few ssds in a high speed non redundant raid would be much better.
Why a few? Isn't a single high-end one enough to saturate an NVMe or PCIe slot these days?
> Or a new class of volitile slow ram that has tons of space would be interesting. So more space at lower cost than ram but slower. That would be interesting. Give me a few 128GB sticks please.
What if this "new" class is actually DDR3 and the "sticks" are actually PCIe cards? That's what I've been suggesting.
Density does matter. And 12 individual disk will have a high failure rate than one disk because there is 12x electronics, 12x motors, 12x seals. Also the power needs for 12 disks is higher than 1. And you would need more raid cpu power to coordinate 12x disks.
> Why a few? Isn't a single high-end one enough to saturate an NVMe or PCIe slot these days?
Sure. I feel like you are just arguing for arguing sake here. You get 1GBps from most ssds and ddr3 is around 15GBps or so.
> What if this "new" class is actually DDR3 and the "sticks" are actually PCIe cards? That's what I've been suggesting
If it was significantly cheaper and new sure. Ddr3 is just as expensive as ddr4 right now.
No body running a professional data center wants the unpredictability of used parts unless they have had very consistent and tame histories. According to large scale studies ddr has increasing error rates as they age: https://ai.google/research/pubs/pub35162
If you absolutely have to store 12TB of data and not spend another dollar, more drives are less reliable. In most situations having more drives makes it a lot easier to protect your data, because you can use parity.
> And you would need more raid cpu power to coordinate 12x disks.
A very small CPU cost; the drives are only doing 100 IOPS. And in return you get 4x-10x the throughput.
I think what we're talking about is aggregates, so think about 120 1TB drives versus 10 12TB drives, if that's helpful.
If the failure rate per drive is comparable, then the failure rate per TB is 12 times higher for the the smaller drives. Arranging them in a RAID doesn't change this calculation.
> A very small CPU cost; the drives are only doing 100 IOPS. And in return you get 4x-10x the throughput.
I did stipulate to ignoring any performance difference, so we can assume that those 12 drives are concatenated rather than RAIDed, which I believe is an immesurable amount of difference in CPU load.
In what way? You have yet to quantify this or the "ton of space" assertion. I'm not saying that density can't matter, just that it doesn't in the practical case. The only time I've heard otherwise is from vendors (hardware or datacenter) pushing a high-density product (which was never any cheaper than the less-dense alternative).
> And 12 individual disk will have a high failure rate than one disk because there is 12x electronics, 12x motors, 12x seals. Also the power needs for 12 disks is higher than 1. And you would need more raid cpu power to coordinate 12x disks.
Of course, but these are all calculable costs (although CPU cost for concatenation is de minimis). So, again, do you just not even perform that calculation?
> I feel like you are just arguing for arguing sake here.
I'm just trying to clarify what point you're making when you specify a particular solution, especially a RAID, which carries an associated cost, in the context of a dicussion about the economy of various options.
> If it was significantly cheaper and new sure. Ddr3 is just as expensive as ddr4 right now.
I maintain that new doesn't matter (see below). As for significantly cheaper, if it's recycled (be it already purchased but unused, spares, or working pulls), it would be a form of gratis-free. If a vendor has significant stock of now-undesirable lower-density or lower-speed memory modules (maybe even bare chips), that can result in a steep discount. It doesn't really matter if it's DDR3 or DDR4.
> No body running a professional data center wants the unpredictability of used parts unless they have had very consistent and tame histories. According to large scale studies ddr has increasing error rates as they age: https://ai.google/research/pubs/pub35162
Thanks for that study. I was unaware memory suffered from aging quite that much. That's further reason for disqualifying from recycling parts of unknown provenance (since they could be pulls due to a UE), but such parts would be disqualified anyway
Still, I fear that you're forgetting we're talking about swap here, not main memory. There's much more wiggle room for doing something like hot-sparing, even without otherwise having a RAID, since the performance of the backend storage (RAM) dwarfs the performance of the frontend (PCIe).
> Conclusion 3: The incidence of CEs increases with age, while the incidence of UEs decreases with age (due to re- placements).
This conclusion also suggests that there's a skew toward early mortality, so a pool of used, known-good DIMMs would actually be more reliable (and predictable, based on this study) than a pool of brand new ones.
It was an interesting discussion. I do think there is a market for another type of volitile storage in PCs as long as DDR prices stay sky high as they are now, faster than SSD and slower than DDR4, and much cheaper than DDR4, close to the prices of SSD and with much higher read-write capabilities.
The problem with SSDs is their limited read-write count I guess as their cost are actually sort of excellent these days.
With regards to hard drives I actually follow backblaze's lead, thus it is received knowledge. They tend to buy the largest drives they can that are not premium priced:
> With regards to hard drives I actually follow backblaze's lead, thus it is received knowledge. They tend to buy the largest drives they can that are not premium priced
This is completely valid for comparable use cases (which I think are vanishingly rare.. usually, performance matters). Their data is, of course, invaluable to the tech community at large (and potentially an annoyance to drive manufacturers).
Hmm, Facebook perhaps.
Emedded devices tend to use zswap/zram which [yet] don't support THP, as far as I know. For instance, take a look at zswap_frontswap_store():
/* THP isn't supported */
if (PageTransHuge(page)) {
ret = -EINVAL;
goto reject;
}
P.S. I didn't check the patch set, may be zswap THP support is already in there.
-ss
-ss
Any kind of "serverless" hosting or any architecture where you have a container per user would benefit from this kind of development.
I'm afraid of latency peaks caused by this, too.
No, because what's driving this change is the rise of NVMe rather than disk, which has consistent access times.
None the less I'm assuming this set of optimizations is targeted not only at nvme storage but also 3D XPoint. Intel and Micron have done a lot of work in this space and promise to provide non-volatile, low latency, and high density in a single package. Micron has seemingly abandoned QuantX (their 3D XPoint) but Intel has recently made some huge strides in this area which, if adopted, will definitely push them to compete in this space.
Soon Intel Optane will be available in a DDR4-like package. It will still be closer to disk-like than RAM like and that is why I think this effort to push for large page swapping will be interesting. It being non-volatile also provides some cool possibilities for the future if this tech becomes more mainstream. Imagine being able to provision a portion of your swap that always contains the state of certain programs you care about or certain always-used libraries. Maybe even caching the entire OS in DRAM-like storage so that all boots read directly from your non-volatile swap.
Also, as a very important aside, this is from the patches mentioned in this article.
From: Huang Ying <ying.huang@intel.com>