Hacker News Reader

Comment by habosa | Hacker News Reader

> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune.

I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.

View on HN

There are a few erroneous replies to your comment, re: Windows support for that much memory.

Memory support for versions of Windows:

                           	  User-mode process	   Kernel-mode
                           	    address space  	  address space
                           	  (process bit-ness)
                           	32-bit        64-bit	        all
  Server Operating Systems:	                    	
    Windows Server 2008 R2 	  4 GB*      2048 GB	    2048 GB
    Windows Server 2012    	  4 GB*      4096 GB	    4096 GB
    Windows Server 2012 R2 	  4 GB*    131072 GB	  131072 GB
  Client Operating Systems:	                    	
    Windows 7 (x86)        	  3 GB*          n/a	       2 GB
    Windows 7 (64-bit)     	  4 GB*       192 GB	     192 GB
    Windows 8 (64-bit)     	  4 GB*       512 GB	     512 GB
    Windows 8.1 (64-bit)   	  4 GB*    131072 GB	  131072 GB
  
  
  * requires IMAGE_FILE_LARGE_ADDRESS_AWARE flag, otherwise 2GB
  ** All units use binary exponents (GB = 2^30 bytes)

process address space != physical memory

Linux (SuSE enterprise something, possibly with a custom kernel, I’m just a user, not admin) runs just fine on 5.2TB machine, though I can easily imagine you’d have problems if you tried it with, say, DOS :)

The system obviously behaves a bit different from a standard desktop machines, e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc., but apart from these, mostly everything seems “normal” to me.

> e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc.

this is NUMA, in case anyone 'new to sysadmin' or to architecture is trying to Google this

I don't believe that much RAM is uncommon for large scale database servers. 384GB RAM is only about $5000 from Dell. They also have a new server model coming out that supports up to 6TB of RAM [0].

[0] http://www.dell.com/us/business/p/poweredge-r920/pd

Most people over estimate hardware cost. Buying a 300-500GB ram server maybe very expensive. But buying a distributed data base expert likely costs at least an order of magnitude higher per year.

Same server in AWS, it'll cost $50k+/month for the same thing. SO just bought it one time fee for probably around $30k or so, I'll say they are doing something right!

The don't have servers with that much RAM, but a 244GB RAM server with 8x800GB SSDs and 32 cores costs <$5k a month ad-hoc and about $1300/month with a 3 year commitment.

$1300/month w/ a 3 year commitment is $46.8k over the lifespan. Which is more then 4x more then the actual hardware cost. Your ROI on a physical server with the same stats at that price tag would be 8 months.

Not saying its totally worth it, but the upside is that they take care of power costs, physical space costs, networking, and if the hardware fails they'll put your application on a new server. Basically you take all of the risk out the equation.

Sure, but add power and cooling for these things plus physical space? (I remember when these big db's needed the floors in the data centre reinforcing since they were a > metric ton )

Still, I agree re. the difference in price. I'd buy too

Worth noting: AWS also doesn't have Enterprise SQL available, so plan on buying that yourself. That's a not-so-insignificant piece of the overall cost to many people.

How common is it to buy one extremely powerful database server? Wouldn't you need the high availability properties of something like a Galera cluster? Or are most people accepting the single point of failure at huge database server?

(I honestly don't know, it's just surprising.)

While you do want HA you also want to keep things simple. You get great HA with just one backup machine. The times you lose two machines are so few that you might crash the site with software failures far more often anyway.

Databases have inherent locking problems that takes more resources to resolve with more machines, more so in some database systems than others. When scaling you often hit a point where you get down to macro logistics, so imagine a car highway: You can get more throughput by adding more lanes. But what that all cars are going to merge to one lane at some point in the middle of the trip (because of a tunnel, bridge or something: it's impossible to have more than one lane at that point)? Now you won't get any throughput benefits of the multiple lanes after all! Just more latency because you get queues up until the single lane and because of the queues all cars are going slow and need to accelerate on the single lane, so the average speed is low too. You are also having too much resources after the single lane because you can never fill all those lanes. It might be better to have one beefed-up single-lane road all the way that people can go fast on. Basically: Remove locks and you get better overall performance.

Yes, this is on the expense of HA. Yes, the costs of scaling up grows asymptotically faster than the costs of scaling out. So this is definitely a trade-off in some sense.

They do have a cluster: http://www.brentozar.com/archive/2012/09/microsoft-sql-serve...

They just run everything through one live database all the time under normal conditions. Personally, I think this is much simpler to manage than to be regularly spreading out queries over many queries.

Windows Server handles that much RAM fine, and, so long as it fits, SQL Server will end up caching the entire database in RAM.

If I remember correctly they're using Dell R720's for their SQL Server machines. At current Dell retail pricing 384GB of RAM would be about $5,900.00.

At that price I'd vote for throwing money at raw hardware and reaping the benefits of caching the database in RAM, versus trying to pay programmers to come up with clever techniques to deal with having less RAM.

It's not a server, but I've seen machines used in physics simulations have something on the order of 256 GB RAM in them. As far as I know they cost maybe $20k-$30k, so not absurdly expensive.

I don't think addressing that memory is a problem on any x64 architecture. Windows Server 2012 has a 4TB upper limit on RAM, apparently [1], and as far as I can tell there's no reason for this other than product differentiation between different "levels" of the operating system.

1. http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

$4k is all you need to build a whitebox 256GB machine: dual-socket G34 motherboard with 16 DDR3 slots lke the ASUS KGPE-D16 ($500) + two Opteron 6320 (2 * $300) + sixteen 16GB registered ECC DDR3-1600 (16 * $160) + chassis/PSU ($500) = $4160.

Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months.

A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no funding, and you are good at assembling and troubleshooting computers, and are ok with mere warranty on parts as opposed to full vendor support from HP/Oracle/Dell, go for it. That's how many startups started (see Google and their makeshift half-donated half-loaned boxes in 1998).

Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.

AWS has among the highest bandwidth costs you're going to run into, if we're talking co-location, dedicated, or self-hosting. Their bandwidth costs are the sole reason I won't go anywhere near AWS yet, I consider their prices outrageously high.

10tb = roughly $1,000 per month

They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.

You can negotiate with AWS.

I don't believe that startups in their garage days have anywhere near the necessary leverage to negotiate anything with Amazon...

Agree, not many people actually realize just how slow and expensive AWS actually is. Kudos to SO for going with their own hardware.

they negotiate, still. I know several startups which went to amazon, said, "hey, your competitor is cheaper, what is your value prop" and suddenly had 50% discounts, etc. It's amazing how much you get simply asking!

Even before OS'es supported that much people were sticking that much RAM on expansion bus cards and plugging them in and treating them like fast disks. So it is going to happen either way. The performance boost for the DB is just too much compared to it thrashing disks constantly and there are quite a few benefits to having fewer machines compared to many DB servers as well.

384 GB is fairly easy to achieve today with 16 GB DIMMs x 24 slots (many general purpose servers have up to 24 DIMM slots). Beyond that you need to jump to 24 GB DIMMs which are fairly expensive.

SQL Server only reads data from memory. When something is requested it is first looked for in RAM (or virtual) and if not found is copied from disk to RAM.

The more RAM you have the better without exception here.

I believe the idea is to hold as much of the DB in memory as possible, for speed, so huge RAM totals on DB servers are not unusual. 384 GB should be well within modern OS reach. Once you're computing at 64 bits, the next theoretical limit on addresses would be somewhere around 16 million terabytes.

edit: wrong number

Not that much, we have 128 gb ram on our hardware. That is pretty standard for upper midle class rackmounted servers.

64 bit computers has a memmory limit so large that it is not understandeble.

The ram is still lightning fast compared to other stuff.

384 gigs is pretty common these days. With Windows, you need at least the server edition to address that much, but if you're running a server, you're not gonna be using the desktop versions of the OS anyways. Pretty much everything else, it's just plug and play. The RAM is quite a bit if you're doing a typical cloud deployment, but SO seems to be more in the model of making sure all your eggs are in a really strong basket. Besides, at the end of the day, RAM tends to be a lot cheaper than lag.

*everything else is plug and pay

Not true. Windows 8.1 64 bit would be happy with that much.

The simplest rule of computing is, if you have a problem that can be solved economically by just throwing more RAM into a single machine at it, you should do it (instead of any other dimension, and assuming you don't have other constraints).

No, that's no common for production servers for popular web services, though- they tend to shard across other dimensions because the web services are often CPU or network-bound.

You can get servers with 384GB of RAM now from a vendor like Dell, without needing to call them for just under $14k.

Their SQL Server is Microsoft SQL Server running on (shock horror) Windows.

Says in the article:

Windows 2012 is used in New York but are upgrading to 2012 R2 (Oregon is already on it).

To be clear, most servers are moving to 2012 R2 (most already have) but the machines our SQL clusters run on, specifically, will not.

This is because you have to effectively (or literally) rebuild the Windows cluster from scratch and we just don't get that level of benefit from the 2012 to 2012 R2 upgrade. There are quite a few improvements we care about: native NVMe, better dynamic quorum, better DSC support, better SMB, and such...but not enough to make the upgrade worth it.

Nick Craver - Stack Exchange Sysadmin & Developer

Do I get this right? You can not upgrade a server cluster to the next version of the operating system without complete new installation??? Could you please write something about these kind of things, this is very interesting!

I'm not sure a full writeup would have much more detail, but yes. With Windows clustering (on top of which SQL 2012/2014 Availability Groups are based), you can only have a homogenous OS version across the board. Since they all must have the same version, you can't have something like a single 2012 R2 instance in an otherwise 2012 cluster...and since you can't do that, you can't upgrade one at a time. Yay! New cluster time instead!

Trust me, we're bitching about this as are most people and I think changes must be coming there. It doesn't matter how many fancy features you add to the OS if we can't upgrade to it, so they'll have to stop and address that problem.

Reply on news.ycombinator.com