Comment by ChuckMcM | Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

These are going to change things for a lot of reasons, not the least of which is significantly higher transaction rates in systems. In particular being past a memory barrier means that your cache is consistent and persistent across reboots in nanoseconds rather than milliseconds or even microseconds means large transnational systems can do more operations at up to 6 orders of magnitude faster. In file systems, and especially so called 'IRON' or error corrected systems, this will significantly boost the speed of operations as more writes can be delayed safely to insure that dense spinning media is writing sequentially rather than randomly. Using the old NetApp cluster architecture I expect you could get a couple of gigabytes per second of random (from the clients) R/W performance on a very reasonable number of drives.

Of course it also makes it possible to rapidly reboot into an OS if all you need in the boot loader is to copy a gigabyte of 'ram' from one place to another and then jump to it. What that enables is powering down servers that aren't being used with the ability to power them up in milliseconds rather than seconds. That has been of Google and other cloud providers to have 'power proportionate computing' clusters.

Very cool stuff indeed.

> if all you need in the boot loader is to copy a gigabyte of 'ram' from one place to another and then jump to it

I suppose you don't even have to copy it before you jump to it. You could map the pages and then lazily copy when you get page faults.

> powering down servers that aren't being used with the ability to power them up in milliseconds

Still, if this is your goal, can't you get 90% of the way there by keeping some DRAM powered up while everything else is off? Refreshing DRAM requires some power, but not that much from what I understand (as a software guy).

Novice in these contexts but I think it would be less of a saving as DRAM must be refreshed, at a pretty high clockrate too, typically.

DRAM has a self-refresh mode powered by an internal clock that you use for low-power standby, so this is not a problem. The DRAM just need to be powered.

> I suppose you don't even have to copy it before you jump to it. You could map the pages and then lazily copy when you get page faults.

You could also just map it and not even fault on access. For data that's mostly read and seldom written, and not too much of a hot spot, there would be no reason to move it to DRAM. Depending on the read latency of these DIMMs, that might include a lot of executable code.

On keeping DRAM alive, at NetApp where the company built custom motherboards specifically for filers, this continued to be an issue for the memory controllers in the processor chips. Neither AMD nor Intel had any support for staying clear of the DRAM configuration on power up of their server chips. There was an interesting config in one of their laptop chips but no ECC on that chip so it was a non starter. At Google the motherboards were less customized and even less likely to incur a Google specific cost of something like holding up memory. The benefit of this coming from Intel is that they do the work to qualify it and everyone else gets to reap the benefits.

The memory bus on these processors is 3 - 30 GBs so copying a full gig would be a 3 - 30mS sort of deal.A pretty short start time. And yes, you tell the BIOS to skip POST (its a common start up option in even off the shelf BIOS packages these days) in order to get a faster boot time.

Netapp can also get rid of their NVRAM architecture. More importantly any journaling file system can have in memory logging now and replication means just transfer the fs transaction logs.

> The memory bus on these processors is 3 - 30 GBs so copying a full gig would be a 3 - 30mS sort of deal

This should say 30 - 300ms I think

> keeping some DRAM powered up while everything else is off?

STR suspend to ram, available in off the shelf computers since ~1995

At the current rating of 30 drive writes per day if you try to use it the way people use RAM now it will maybe last a week if you're lucky.

It is important to remember that this isn't FLASH. It is more like FRAM with its non-volatility. When Intel and Micron announced it three years ago[1] they made some pretty bold claims, like 1000x faster than NAND, unlimited write endurance, Etc. For a long time it was just vaporware and then it came out as "NVME" type SSDs and now they have figured out how to put it right on the frontside bus with the memory controller.

[1] https://newsroom.intel.com/news-releases/intel-and-micron-pr...

not fsb, memory controller is now integrated into the processor die.

Ah good point, they are still north of the cache and MMUs though (as opposed to being on the PCI bus and behind the IOMMU at best)

Of course, it comes at a cost: These dimms have several orders of magnitude higher latency and a fraction of the throughput of DRAM.

Once you introduce volatile memory as highspeed buffer, the coolness factor drops quite a bit as you end up in the same volatile/non-volatile tiered storage split that we have always had. It's just back on a parrallel bus like in the good ol' days of PATA.

How do you know? Other comments have pointed out that Intel was aiming for Optane DIMMs with a read latency half of the current RAM.

I appear to have exaggerated slightly when I said several orders of magnitude, but the original 3D XPoint latency numbers put it at about 25 times higher latencies than contemporary DRAM (~7000ns vs. ~200ns).

Likewise, quad-channel DDR4 setups can push around 60GB/s, while Optane PCIe SSD's have only been able to push around 2GB/s. The PCIe x4 interfaces that they use should have been able to push 4GB/s, and if they started hitting a wall there, they could just have used a full PCIe x16 interface which could push around 15GB/s. This indicates that the throughput was not interface bound.

Nothing seems to signal that Intel have have anything up their sleeve to provide the 25x latency improvement and 30x throughput improvement needed to just be even with DRAM, although placing the chips on the memory bus will of course provide some speedup.

Note that transactional costs on PCIe are much higher than they are for memory, and you are going through a disk driver (if you're and SSD, you can avoid that using the NVME architecture extensions) There is no driver needed for non-volatile storage on the memory bus. You just access it like memory.

I would not be surprised if early users of these devices were seeing 10GBs+ of data throughput.

PCIe 3.0 does not have a very high transaction cost. The encoding overhead is just 1.5%, and the protocol is quite simple for the primary communication (there's a bunch of "extras" like retransmits and stuff, but that's not important here).

I can say that the PCIe x16 cards we develop at work have no problem reaching the theoretical maximum, transmitting around 15GB/s worth of payload data (e.g. our 2x100Gb/s NIC with one port pushing 100Gb/s in both directions, and another a few tens of Gb/s in both directions). We don't make cards smaller than x8, so I can't give measured numbers there, but Intel should have no problem transmitting ~4GB/s on that x4.

NVMe surely has an overhead, but this overhead is far from interesting, unless you are implying that the overhead eats a whopping 35% of the available bandwidth. Likewise, if Intel had hit a PCIe/NVMe bottle-neck, going to up x8 would not not have been difficult in any way (x16 is more annoying, though).

The performance has either been restricted by the flash itself, or by the controller—not by the interface. The numbers are also small enough that the software side shouldn't be a problem yet. Attaching to the memory bus minimizes penalties that are unlikely to be a bottle-neck.

for rebooting, you'd have to have super-fast POST and device initialization after power cycle, so i wouldn't say milliseconds.

Definitely. Most of my Dell machines take 2-5 minutes to get to the "booting OS" part of the process, but a couple of them have at times taken 3 hours.

The latter is probably a firmware bug, but the best Dell ProSupport has been able to do to solve it is "flea drain" (unplug, hold the power button for 60 seconds, plug back in).

Dell servers are notoriously slow to boot, but this is due to the remote management subsystem being extremely slow.

It's quite a pain in the arse, and one of the reasons we prefer to use Supermicro machines for development at work, using only Dell for testing.

At least we've been moving glacially in that direction, thanks to UEFI & faster booting. POST, device initialization, etc has begun to stick out, and is slowly being improved.

He doesn't mean rebooting will take microseconds/nanoseconds, he means the transaction commit will take that long. Before, a transaction commit meant flushing disk, at least as far as the disk controller. Now, commit = memory barrier.

"What that enables is powering down servers that aren't being used with the ability to power them up in milliseconds rather than seconds." (great-grandparent)

Actually and "restarting". Technically it isn't "booting" because the OS is already in memory and initialized, it just isn't running.

Given your typical motherboard which you tell to bypass POST, it can go from power applied to memory controllers alive in just a few microseconds. The code then to go from there to running is also quite small, especially if there aren't things like graphics controllers on the PCI bus that have to be initialized first.

And even those can probably be lazy-loaded, with a page fault in the IOMMU and handled as soon as the rest is done, or the rest isn't doing anything and something is thus blocking on the GPU initialisation.

You can write your own custom firmware and skip the unnecessary bits.

The aerospike demo Intel did at the event showed 17 seconds vs over 2000 seconds for a restart. The difference being rebuilding a dram index on a many many terrabyte data set. Remembering aerospike has consistency, so you can't have the index be slightly wrong.

close your laptop lid, wait 10 seconds, open it up, did it resume quickly?

2017 MacBook Pro 15". No, it doesn't. Open the lid, time passes before screen is no longer black. Even longer amount of time before the TouchID figures out my finger is the accepted finger. So, when did it resume? When the screen lit up, or when I could actually use it? Obviously, Apple's TouchID is preventing me from resuming, but to the user, that's all they care about.

You made the choice buying Apple.

Reply on news.ycombinator.com