Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

I just boggle at the idea of a server with 1TB of RAM. I'm sure the oil&gas folks are salivating as this allows them to to put an entire high resolution 'cube' into memory and analyze it but for us mere mortals at what point is there so much stuff in memory that you need a couple hours of hold up time just to flush it out to SSD.

View on HN

The big headache is not 1TB but 64TB which is the maximum physical RAM limit of the Linux kernel / x86-64 architecture. Big NUMA systems could go higher but they don't. Look here: https://www.sgi.com/products/servers/uv/ "SGI UV 300 scales from 4 to 64 sockets with up to 64TB of shared memory in a single system." vs "SGI UV 3000 scales from 4 to 256 CPU sockets with up to 64TB of shared memory as a single system." see how 256 and 64 sockets both only support 64TB?

More ordinary servers typically stop at 6TB since one Xeon can support 1.5TB so an ordinary quad socket board typically with 96 DIMM slots can go up to 6TB with 64GB DIMMs. You can configure a machine like this at http://www.thinkmate.com/system/superserver-4048b-tr4ft and see that for the relatively low price of $110K you can get a machine with 6TB of RAM.

> The big headache is not 1TB but 64TB which is the maximum physical RAM limit of the Linux kernel / x86-64 architecture.

It sounds like you know what you're talking about, so I'm sure it was inadvertent that you wrote "physical RAM limit of the Linux kernel". It's primarily the x86-64 architecture and 5-level paging is coming which extends the linear address space to 57 bits (128 PiB) and the physical address space up to 52 bits (4 PiB). Still, one wonders how long it will take for that to be inadequate.

https://software.intel.com/sites/default/files/managed/2b/80...

And Linux support for 5-level is being actively worked on since December, a quick git search shows this in 4.11:

    Merge 5-level page table prep from Kirill Shutemov:
     "Here's relatively low-risk part of 5-level paging patchset. Merging it
      now will make x86 5-level paging enabling in v4.12 easier.

There are still 8-socket systems, like the Supermicro 7088B (192 DIMMs), which support a theoretical 24 TB (with 128 GB modules).

(It would be interesting to know whether this backplane-based system actually manages to achieve 4.8 GHz QPI speed (9.6 GHz symbol rate), or whether the physical aspects limit it to lower speeds. Given that processors only have four QPI links this would further increase communication overhead -- in a 8-way system some sockets are separated by two hops).

So 128GB modules exist and available http://www.memory4less.com/crucial-128gb-ddr4-pc19200-m386aa... but at least 30% more expensive than 2*64GB: http://www.memory4less.com/samsung-64gb-ddr4-pc17000-m386a8k... and of course this is the cheapest vendor. Brand names will fleece http://www.memory4less.com/hp-128gb-ddr4-pc19200-809208-b21 you for 128GB modules while http://www.memory4less.com/hp-64gb-ddr4-pc17000-726724-b21 64GB is just a little more expensive.

You can buy 256GB of DDR3 RAM for just $576, so 1TB is only $2304. Example:

http://www.ebay.com/itm/182537878712

Of course that's just RAM. If you want a server with it, it's not much more, just over $1K shipped:

http://www.ebay.com/itm/391697886461

It's true, 1TB of memory is well within the realm of an over-eager enthusiast. Motherboards that support it aren't even exotic, they're just physically large server-grade ones.

Completely. 2000CUR is nothing for someone who is looking at this. It's an apple laptop. Nothing smashing.

That said I still want my OS to fit on a fdd and have 64k demos generating the universe.

They are often based on lower density DDR3 chips that consume more power per bit. That being said you can do 1TB on a normal dual socket server with 64GB DDR4 LR-DIMMs for a while now, though they do come at a cost per bit premium.

We've had database servers with 2-3TB of RAM in production for years. You can get servers today with 24TB RAM - they're 8-way so lots of NUMA going on but still tons o' RAM.

> at what point is there so much stuff in memory that you need a couple hours of hold up time just to flush it out to SSD

A few hundred megabytes for our "enterprise" financial software...

  mere mortals at what point is there so much stuff in memory that you need a couple hours of hold up time just to flush it out to SSD

If your buying these machines with multiple TB of RAM, then buying flash arrays that can drive multiple GB/sec of IO bandwidth shouldn't be a problem either. At say 8GB/sec write bandwidth flushing 1TB is a little over two minutes. Although, why you have that much "dirty" data in RAM might be another question. A database machine using that amount of RAM is going to care more about the IOP rates of the disk, so that its flushing updates to disk at the same rates they are arriving. Meaning that the RAM won't need to be flushed to disk if the machine/power/whatever fails. Disk arrays with >1M IOP/s have been around for over a decade, and given a SAN can be wired together to increase aggregate performance.

Run your whole system in RAM with periodic backup to "slow" SSD?

> I just boggle at the idea of a server with 1TB of RAM.

MooseFS (a scale-out storage system) keeps metadata in RAM for low latency and is known for pushing the limits of commodity hardware in big installations. I presume the same applies to any kind of low-latency database of similar size.

1.4 gigabytes per second or more for 4K flat video work at 4 color components at 16 bits at 25fps (PAL land) each raw for each channel on the timeline saturates system real quick. For online finishing systems that would be sweet. 8 TB would fill an average 90 minutes movie if you had a finishing timeline (no additional ones over it). With over 12TB I wouldn't know what to do anymore, as of this point in time.

Sure, you can build a RAID that fast already though.

The bread-and-butter HP DL380 goes up to 3TB. On the other end of the spectrum you have beasts like the IBM E880 that goes up to 32TB.

The SPARC M7-16 has up to 16 TB of ram, sixteen processors (512 cores, 4,096 threads), 48 PCIe 3.0 x16 slots:

http://www.oracle.com/us/products/servers-storage/sparc-m7-1...

This is just the mainstream catching up.

How does it benchmark against x86 systems available now though?

I was under the impression SPARC was on life support

Those of us in the Big Data space can easily chew through that.

Few HDFS data sets stored in RAM or some more Data Science Spark workloads.

Try an analytics / reporting database server. Lots of data to keep readily available, but almost no dirty pages.

I guess I should have been more clear and said a 'commodity server' with 1TB of RAM :-)

I've worked with companies running MySQL with a terabyte of ram. Fun times.

Reply on news.ycombinator.com