Bringing SSD Performance to the DIMM form factor
sandisk.com.br
sandisk.com.br
Energy consumption, heat dissipation, all of this becomes a larger problem for smaller structures. It's just easier and less error prone to build larger structures, yields are higher, so costs are lower for larger structures.
I'm certain the tradeoffs will change, but it's all limited by some physical law and while we can sometimes trade one law against the other, we can't currently break them, so tradeoffs will always exist.
(That “currently” is just a weasel word, but this being the Internet I understand why you put it there)
It is highly improbable that this new technology cannot be optimized in any relevant metric by trading off optimality in a different metric (this is what you are actually claiming).
The laws of physics dictate what the ultimate limits of a techology are; they do not dictate which technologies you actually choose to use.
I never said so, I agree. I'm just stating that I'm confident that in the future, we'll still have tradeoffs. And I'm certain those will include "fast and small, but expensive" vs. "slow and cumbersome, but cheap".
We already reached the point where it's a problem that the clock signal on a die spreads so slow that it's a significant problem to increase the clock rate. So we're pushing parallel computing and other solutions, but those come with their own set of trades to make.
http://umumble.com/blogs/hardware/machine-perception-of-time...
They are, they're the laws of physics. According to known theories information cannot travel faster than the speed of light, so unless a very very radical new discovery is made sometime in the future, size and speed will basically be inversely proportional to each other.
Just look back at how things were before the discovery of GMR[0]. We had very different tradeoffs then as compared to now.
Which technologies we use for our caches and main storages will change, but we're never going to get around the need for caching.
It'll be a huge paradigm shift in storage and IO speed.
It's unlikely that future technological advancements will give malware authors more tools to work with? That seems unlikely, if history is any indication.
Imagine when you have 1TB of non-volatile storage which operates at speeds close to those we currently enjoy for DDR3 memory. Most average consumers won't use all of that space. It seems entirely plausible that a virus will write itself to some area that is never overwritten. It becomes a latent piece of malware, invisible to any filesystem, but activatable on demand by some other hard-to-detect malware. This two-step scheme would be difficult to detect since the second component does nothing except activate the first component (stored in nonvolatile storage) under some specified criteria.
Or, how about key generation or password input? Remember cperciva's post about how it was extremely difficult or even impossible to zero a memory buffer correctly? Now add in the idea that suddenly "everything is nonvolatile" and you get a perfect storm for security concerns: if a program segfaults or is terminated mid-operation, then suddenly any sensitive data it had spit out into memory might be capturable by other programs. Or, since it's nonvolatile, those secrets will persist "forever" (until they're overwritten, which may be a long, long time if storage is plentiful) and you may inadvertently leak secrets when you sell your memristor-backed storage. Whoever buys it can do some digging and uncover whatever was written into that buffer, if it's truly non-volatile storage.
So, while the future is exciting, underestimating the security implications probably isn't a good idea. We can develop new methodologies to combat the new security concerns, but new security concerns will almost certainly come to fruition. Of course, calling the concerns "new" is somewhat unfair, since new concerns are almost always very old concerns manifesting themselves in new domains, but new domains usually enable malware writers with new toolsets to exploit.
True, even under this model, you could probably have a virus running "resident" in a core without the OS even being aware of it. Basically, it's be like the MMU was a hypervisor and the OS was just a domU within it, unaware of viral "domains." But those viruses probably wouldn't be able to do much, except waste electricity, because they'd need to either get the IOMMU to allocate them some hardware to do IO, or get the OS (which has such hardware) to agree to do IO on its behalf. (Though now we can get into things like intermittent side-channel attacks, where you can mine bitcoins on one cell for minutes and then only have to get the OS to leak a single packet when you have a (rare) success, which is much simpler than getting a whole stable TCP channel or something.)
> 1TB of non-volatile storage [...] Most average consumers won't use all of thatWhen we're creating fully immersive 4D environments in place of holiday snaps then I imagine we're going to be using quite a lot of memory waving through our holiday review with friends.
Funny, I'm the opposite. With all this bandwidth available, why keep stuff?
Locally, if you're using RAID like everyone else, you just have to trust your local electricity supply and your ability not to rm -rf /.
Remote is fine for backups, though.
The 'enables new opportunities that were previous infeasible' component is more exciting. But generally, I think most of it gets swallowed up in more of the old.
The only thing worth debating is whether the GUI design pendulum will have swung between Flat/glossy an even or odd number of times by then :p
I'm sure people will find something to do with it but I'm having a very hard time thinking of situations where you want massive amounts of RAM and also need to be very power conscious or wary of power loss. Certain types of database, perhaps? I wouldn't imagine HPC worries much about power loss. Video game textures don't need to be nonvolatile.
For most purposes you can keep shoving in DRAM and SSDs until you have enough space.
Yeah, of course, everything has security implications and some technologies even more so, but unless one explains them and states how they are specific to the technology being discussed (like you did, for example) then the comment doesn't really add any value.
Writable storage using a centralized access method (file system) is hard enough to purge and keep clean, having a memory mapped persistent chunk of storage that is user process level writable creates an entirely new class of problems.
Getting rid of viruses from BIOS flash is hard enough, this expands that concept to all of userspace.
I sincerely hope the 'x' bit will be off for such memory.
A simple way to counter this is to tag the memory pages as free/used. If the memory controller reads a page as free, any reads from it can return zeroes without hitting the actual memory bus. Any piece of code that wants to hide will not be part of any active object and its page being marked as used should be considered an allocation error and be easily detectable.
Used to get them regularly on my Windows 3 -> Windows XP/2000 days.
Haven't seen them much in Linux or OS X in the last 15 years.
There might have been one or two. But I doubt you can find 10. And the number of users I've seen reported as affected by one was also laughable, something like 1% of all users of less (half to around 1 million).
The non-hacker media of course always touts this or that trojan that actually needs user intervention to be installed as a "virus".
I just assumed they are integrating it into their new super computer
For ordinary computing, maybe a terabyte of flash will finally be enough.
I would be willing to wager that new models of laptops in 2020 with spinning disk will be uncommon, and, by 2023, except for niche verticals. basically non-existent. Same for Desktop 2 years later in each category.
2025 will mark the end of consumers buying spinning disk.
How about more like 2015. You're already forgetting MacbookAir/Chromebook/Ultrabook crowd are slowly taking over (which have no spinning disks).
Problem with Flash is the more you pack in same volume the more of it comes out bad, while the rest is fragile ticking bomb.
Replacing slow, but reliable HDD with 100 writes limited flash is not the answer.
Of course, all of this is pretty marketing heavy and not a lot of details have been released, so take this with a huge grain of salt. HP's CTO wrote a fluff piece on flattening the storage hierarchy here: http://www8.hp.com/hpnext/posts/end-necessary-evil-collapsin...
[1] http://arstechnica.com/information-technology/2014/06/hp-lab...
[2] http://www.businessweek.com/printer/articles/206401-with-the...
Inventors of anything submitted in patent applications should be named on the application. That is a potential route to finding who in a corp actually invented stuff.
(source: I'm a cs professor at carnegie mellon, and I was just chatting with some of those researchers at OSDI.)
Real revolutions in OS tech have for the last decades come from the keyboards of large institutional users and from hordes of smaller ones. Even IBM has pretty much acknowledged that. So we'll see if HP will be able to keep the door locked to someone doing a linux device driver for their new hot hardware or to add a few calls to the kernel that enable an application to use this tech.
The same has happened to infiniband (fast networking for clusters and storage) and tons of other esoteric hardware including shared memory between physically disjoint machines (DSM).
Still they're merging entire sub-systems that have existed independently for decades and someone has to actually make all the changes to support permanent in-place execution of code and the like that's what I mostly got out of the OS porting news.
And e.g. consider the changes to RDBMS's if you can do guaranteed committed writes of individual bytes. A huge amount of the work on RDBMS's have been around table types that are optimised for per-block read/writes, where the design tradeoffs might be very different if you can do "cheap" random access (though caches will probably still justify some degree of block-wise access).
But with respect to operating systems, I agree that it will take a long time to see any dramatic changes. The current mechanisms would still work: Open a file that maps to a memory region instead of a drive; mmap() it. Your app would just want to know whether or not the file content is instantly persistent on write or not.
I think HP's starting point, though, is that we don't really know. Both in terms of hardware and software. Take this DIMM drive, for example. If you look at current server motherboards, a lot of servers has no need for PCIe slots other than for SSDs. A lot of them will not need drive slots at all if you can stuff that much storage on DIMM slots. Suddenly it becomes viable to put really high performance systems in half depth 1U form factors. For a lot of our larger servers, 50% or more of the server volume is air because current designs are so geared towards front loaded hot-swappable drives. Maybe 75% of the weight of the servers are due to drive bays and larger boxes to accommodate that front-loading. That's a lot of wasted metal, and a lot of wasted rack space. And a lot of wasted electronics for SATA interfaces that could sometime become obsolete.
Unless your writing at the machine level I really do not see the need for most programmers to ever concern themselves with storage, that should be the work of the underlying machine code and/or OS.
I really recommend Frank Soltis' Inside the AS/400 if you are interested in computer architecture and virtualization.
Aren't the latest versions of PCI-e SSDs already approaching early version DDR3 transfer rates?
I do have a hard time picturing what this will look like if it was as fast as traditional RAM. If I can store everything in RAM, from the OS binaries, to the running processes, it certainly has a kind of elegance to it. Lots of microcontrollers already act this way: all your hardware has an address in a single address space, be at a hardware port, ROM, RAM, NVRAM, etc. However, I wonder how the modern UNIX OS will work with a system like this without block devices at all. I wonder if what it'd do is actually just use RAM disks, and years from now we'll still be doing this the way we do double emulation of the TTY. That'd be kind of sad since it means we can never get away from the concept of old block devices.
On the other hand it's damn convenient to be able to just append to a file and not have to worry about reallocating it. This means that perhaps all we'd need is a filesystem that's designed to work over RAM rather than over block devices.
We run clusters of ephemeral linux hosts, and while we use disk, we also have plenty of hosts that run diskless. The disk mount point just becomes a tmpfs mount and the required minimal installation is made on a tmpfs partition which amounts to a trivial amount for hosting busybox and some conf files in /etc. Linux does appear to understand that tmpfs backed filestores are already in memory and do not need to synchronize pages and flush out dirty pages. That whole caching/memory layer for this fake block device is just not used at all.
I see two possibilities:
- Two separate address spaces, one volatile (RAM) and one persistent (SSD). "Writing" to persistent storage is as simple as "mov [src],[dst]".
- One global address space, with a page attribute of "volatile" (true/false). Writing is just as simple as above.
At that point the OS and userspace are free to create whatever abstractions are necessary on top. They don't necessarily need to be filesystems or files per se.
Like http://ddrdrive.com/ ?
I didn't think there was any shortage of these kind of devices - especially given their common use in ZFS workloads as a SLOG.
I don't think that's how it works - I think you get a special driver that presents the fast-storage-in-dimm-slot as a block device. So you still have a block device, it's just faster.
Right ?
Didn't Palm OS originally work this way? The early devices were too cheap for flash memory for anything more than the base OS, so they just stored everything in battery-backed RAM. If your batteries ran out, your device was wiped and had to be re-synced. Apps were run in-place in RAM. Then they had to hack in a bunch of workarounds when they added flash memory storage.
So SanDisk is likely quoting a worst case read latency and/or a best case read IOPS. Customers are left to themselves to figure out which of these numbers is most likely to represent the average performance...
PS: here is a paper with some more details but that still fails to explain this discrepancy: http://www.snia.org/sites/default/files/SanDisk%20ULLtraDIMM...
Maybe you can submit more than 8 queued requests to the controller and have them stream back at the full DDR3 data rate. Maybe it's not actually addressable as RAM, it just uses the DDR3 interface as a communication bus.
https://en.wikipedia.org/wiki/Delay_line_memory#Mercury_dela...
and a video interview and demo with a SanDisk manager at a computer fair - http://www.youtube.com/watch?v=jarsTLGXx9c (currently only available for OEM, requires special bios setup)
So about 4 unrecoverable errors per sector. Seems legit.
Do people not read their own copy?!
Source: http://www.anandtech.com/show/8396/fms-2014-sandisk-ulltradi...
TL;DR about 4x faster then PCIe SSD's
http://www.percona.com/blog/2014/08/12/benchmarking-exflash-...
So the 95%, write latency is 1-1.5 msec for one model, and 2.5-3.5 msec for another model. Maybe representing SLC/MLC flash?
95% read latencies are maybe 350 or 250 usec.
Like an SSD.
> Is it just using the memory controller as a super fast parallel bus?
Afaik, it's this.
DDR4 directly supports 512GB, and also throws away three potential column address bits.
"nvDIMMs" on the other hand do look like and are as fast as plain memory. Disadvantage is that they are not big.
And in most servers these days, drives take up a massive amount of physical space. Being able to drop the drives and RAID controller has the potential of massively simplifying some server hardware.
But I thought the main drawback of SSDs was that eventually the individual memory cells will degrade and lose the ability to write new data. I don't see anything about endurance other than a MTBF of 2.5 million hours and the disk-writes-per-day (DWPD), which I had to look up. I have no idea if these are good or bad. I feel like these things will be great if you didn't write new data a lot, or you have deep pockets to replace them when you need to.
On any kind of large-scale installation you need redundancy, but if those numbers pan out then you could expect any single SSD to outlast your organisation.
Take backups anyway.
I think the market for this could be much bigger if it behaved like a regular RAM DIMM, only slower and nonvolatile; it somewhat reminds me of old machines that used magnetic core RAM. This could be useful for laptops, like a zero-power suspend-to-(NV)RAM. The only thing that is worrying is the endurance of the flash - especially if it's being treated almost like RAM in this application.
For comparison, the stated read performance is 800MB/sec or so. Last time I checked - on ca. 8 year old hardware, we could read files from a RAM disk under Linux at a rate of about 6GB/sec. That's with the added overhead of the benchmark software running and going through a filesystem.
http://en.wikipedia.org/wiki/PC133
I also wonder if that 880 is for just one slot filled up. If it is, then I wonder if you can parallelize to 4 slots for ~3.4GB/s.
At that point, I imagine many types of data sets would work A-OK.
Though I wonder how that's scaled for cores, given that in the situation I imagined above, you'd be streaming all that data to just one CPU to get that aggregate bandwidth.
Hmm.
OK what about some other scenarios? Being able to have RAM and Storage on a sliding partition is sort of cool. I wonder if it'd make provisioning VMs any easier. "How much RAM do you want? Set the slider!"
> If it is, then I wonder if you can parallelize to 4 slots for ~3.4GB/s
You probably can, but then you can't run code straight out of it, or read/write data directly as if it is normal RAM. Putting a filesystem on it, RAID'ed over multiple slots will likely work great (especially so since it's easy to get servers with a ton of DIMM slots, and it'll allow much better server density than trying to fit in more drive slots). But treating it as RAM will be slow.
It will definitely change the way we code
Additionally I suppose your variables could also be persistent automatically. Though the biggest change I see is that software would never really need to completely close anymore when RAM == Disk. For RAM usage it wouldn't really matter if they were running or not, unless they delete data upon shutdown. To save CPU or other IO usage the application could simply be suspended.
I don't think it will work out of the box on regular systems. Also, isn't pcie better for this?
I don't think this SSD can be accessed through regular SCSI, IDE, AHCI controllers. http://www.sandisk.com.br/enterprise/ulltradimm-ssd/
Or flick rapidly between Linux and Windows. That makes me think it might be wise to have a manager to do this. E.g. leave the disks accessible between three+ operating systems.
Isn't that what you already get with suspension to memory (S3)? At least on my Debian laptop, it's ready before the lid is fully up.
- S3 uses battery power (albeit not much)
- NVM provides protection from power faults in datacenters
Of course, if it behaves like a regular SSD, the read speeds still exceed SATA3.0 speed.
Does this mean that for software to take full advantage of this, the software will need to be updated to account for it? For instance, Redis loads data into memory from disk upon starting, but if something like this 400GB SSD is available as memory (and say for instance, 300GB is in use by Redis), wouldn't it make sense for Redis, upon starting, to just "remember" the state of the memory rather than reloading it from disk?
You would not use this to replace RAM, but rather have a second section of memory for persistent data.
SATA 3.2 (2000MB/s) would help for throughput but not for latency. It is also still quite uncommon even amongst latest generation servers, whereas this SSD is compatible with many current and older generation servers.
http://hothardware.com/Reviews/OCZ-CES-2012-Product-Tour-ZDr...
Also available in a non-Brazilian page at http://www.sandisk.com/enterprise/ulltradimm-ssd/, of course. :)
E.g. I can get 1U servers with 32 DDR3 slots and 64 cores, but only space for 3x 3.5" drives, or if you double up (but few chassis will then let you use hot swap caddies), 6x 2.5" drives.
You could easily fit 256GB RAM and still have 16 slots free for RAID arrays over those DIMMs.
Not yet. I think first we will have SSD as static RAM plus dynamic DIMM RAM as Cache. I wonder when SSD will compete with DIMM regarding maximum number of write operations.
Using a different socket is best in all cases.
Try and boot an OS without knowledge of this, and it'll start using the stick like it uses RAM. It'll take orders of magnitude longer to boot, to the point of impracticality, probably (I haven't calculated).
The OS needs to understand the memory map, and at the minimum exclude use of the stick for "normal" operations.
Then, it needs to expose the stick for access - preferably in a way that is more usable than /proc/mem.
Flash still has limited write cycles. Its for data you probably wont write more than once an hour, but not virtual memory.
> Flash still has limited write cycles. Its for data you probably wont write more than once an hour, but not virtual memory.
If you write 20GB of data per day to a modern 480GB SSD you can expect 25 years of service. (Realistically, something else will probably fail before then)
Example - http://www.anandtech.com/show/8520/sandisk-ultra-ii-240gb-ss...
"Reduces total processing time, compared to hard-drive drives (HDDs)."
i loled.