AWS C4 Instances | Hacker News Reader

AWS C4 Instances

aws.amazon.com

50 pointskapkapkap68 comments

In case anyone else was wondering what "ability to control P-States and C-States" is about, I think this is a useful reference [1]. I guess with the C4-8Xlarge you control the entire physical machine and you can enable additional P-states to get more power out of the cpu while using more electricity.

[1] http://www.xenserver.org/partners/developing-products-for-xe...

That's a strange feature to include on a machine I'm not paying the power or cooling bill for. Why would I ever set it to anything other than 11?

From https://aws.amazon.com/blogs/aws/now-available-new-c4-instan... :

  In some cases, your workload might not need all 18 of the cores (each of which
  runs two hyperthreads, for a total of 36 vCPUs on c4.8xlarge). To tune your
  application for better performance, you can manage the power consumption on a
  per-core basis. This is known as C-state management, and gives you control over
  the sleep level that a core may enter when idle. Let’s say that your code needs
  just two cores. Your operating system can set the other 16 cores to a state that
  draws little or no power, thereby creating some thermal headroom that will give
  the remaining cores an opportunity to Turbo Boost. You also have control over
  the desired performance (CPU clock frequency); this is known as P-state
  management. You should consider changing C-state settings to decrease CPU
  latency variability (cores in a sleep state consume less power, but deeper sleep
  states require longer to become active when needed) and consider changing
  P-state settings to adjust the variability in CPU frequency in order to best
  meet the needs of your application.

I question the real world utility/practicality of this, but thanks for the explanation.

To elaborate: I start with (e.g.) a 2 cpu c4 instance. But my single thread performance isn't fast enough. So I'm going to spend a king's random to upgrade to a 36 cpu instance, just so I can disable 35 of those cpus? There are several other options I'd probably investigate first.

Some additional information on processor state control is available in the docs including some sample scenarioes

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processor...

They'll raise the clock speed on a single core if it's cool enough iirc, so better single threaded performance

Ran UnixBench

c4.2xlarge https://gist.github.com/karlseguin/a659ef87b3a4a5d590e9

c4.8xlarge https://gist.github.com/karlseguin/0f38b9ad87ebd54375da

i7-4770 https://gist.github.com/karlseguin/5a6a45ace2048545b6c3

For those who prefer AWS blog posts over the product page:

https://aws.amazon.com/blogs/aws/now-available-new-c4-instan...

Still damn expensive compared to things like a dedicated server from here:

http://www.kimsufi.com/en/

Amazon seems to offer more network bandwidth though, so they might have an advantage for CPU-heavy and network-heavy loads. But from the testing I've done Amazon comes in almost dead last for cost/performance on most metrics. I suppose they could make sense if you make heavy use of all their services, and maybe their cost/performance improves with scale?

Most of the servers on that page aren't even running ECC RAM. The only one listing ECC RAM is the E5504 which dates back to 2009 whereas C4 is running the fastest Xeon model available today. The rest of the list is Intel Atom and i5 and i7 desktop chips. 60% of the server types on that page are also currently out of stock.

Where does EC2 say that they are running with ECC RAM?

here: http://aws.amazon.com/ec2/faqs/

Oh awesome, thanks!

Unfortunately Kimsufi / SoYouStart have a tendency of not always having servers available - either the kind you want or in DC you want.

True. Those are better if you want to grab a server and sit on it for a long time as a dedicated box at a data center. It's more like old school rack hosting but you don't have to actually screw with hardware. They'd be awful if you want elasticity, which I suppose is why "elastic" is in Amazon's name. Duh. :)

I have some stuff at Kimsufi (BHS data center). The performance and stability are really excellent but I'd have to wait for availability to add more, which sucks.

Plus seamless integration with everything else AWS.

You should be aware though that integrating with AWS services that aren't also provided by AWS' competition, you're increasing the cost of switching away from AWS in the future.

I'm not saying don't do it, I'm just saying be aware that you may be making it harder for your company to change providers down the road.

Exactly. I would think very, very, very hard about building something on top of any AWS technologies for which there are no good open substitutes of.

It's a trade-off.

As far as I am aware, there is nothing out there that can rival the cost and functionality of S3. I'm not going to avoid using it simply because there is a lack of open alternative - it's that much better that I will use it now and redevelop to use different technology if and when I have to.

After all, most of these services all have pre-made client libraries. Rewriting code to use a different service isn't likely to be world-shattering.

I think your own cluster of for-rent machines would outperform the price performance of S3 wildly. Using e.g. Riak or something similar.

(I haven't done this particular pricing calculation, but everything else in AWS seems awfully overpriced, so why not S3?)

But then I have to maintain a Riak cluster.

For all means, if you have LOTS more money than time, you should pay someone else to do this. Outside of certain sheltered silicon valley startups, this is often not the case though. :)

You know, I'm pretty sure that almost everyone is better off spending a few cents per gigabyte per month on S3 for inert file storage, a functionality that's performance-inelastic except at the far end of the curve, than maintaining a Riak cluster.

Source: I've maintained a Riak cluster.

There are lots of AWS services I avoid (I won't use SQS, I won't use Dynamo except during prototyping, etc.), but S3 is so utterly portable across any cloud you go to (and latency is already not something you are worrying about if you're using S3) that it's laughable to suggest that you're going to roll your own with anywhere near the reliability at anywhere near the cost.

We rolled our own proprietary version of what became known as S3 and scaled it up to several thousand nodes. At any given time, less than half a FTE spent any time on this, and I think we've had roughly equal (probably better) availability than that of S3, at a fraction of the cost.

Assuming you pay your engineers a paltry $50k/yr, half of a FTE is $25k/yr. Quick calculation based on the public S3 pricing says that's 847TB of data you could be storing just with their salary. That doesn't include all the actual costs associated with running thousands of nodes - acquisition, hardware replacements, utilities and other operating costs, space, etc.

(To be honest, with that many nodes I have a hard time believing that you're not spending half a FTE just replacing dead drives.)

Not to mention the additional costs if you're actually interested in the same level of durability (target eleven 9s (99.999999999%), but last time I talked to an engineer they're still at 100% since leaving beta). Multiple DCs, transit between them...

I'm sure there's a scale at which it's worth it, but it's pretty far up there and for almost everyone S3 is a cheaper option.

I spent zero FTEs on S3 and average around twenty-five bucks, or under a third of an FTE's hourly amortized rate, per terabyte.

You might not be in "almost everyone", but almost everyone else is.

Yeah, we use S3 at the company I work for a little under a terabyte and we pay less than $25 a month.

I think you're overestimating how expensive S3 is. You'd have to be storing a lot of data before running your own infrastructure becomes more affordable (especially factoring in employee costs)

Not to say it never happens, but 90% of people are better off just using S3.

Why wouldn't you take advantage of everything your platform gives you? This goes for your OS as well as your infrastructure.

As far as I know, the biggest reason not to rely unconditionally on AWS-specific offerings is the same reason to avoid vendor lock-in elsewhere: They've got you over a barrel when it comes time to re-negotiate pricing. Amazon has a pretty wonderful history of price cuts, but when Google comes in with a better deal (or you want to hedge your bets with a hybrid cloud-type thing), it's waaay harder to find analogues of things like Kinesis and SES and Elastic Beanstalk.

There are definitely some PaaS-type offerings they have that aren't as scary (RDS/MySQL, for example, and ElastiCache/Memcached, etc.), but if you're trying to bounce systems between clouds, or deploy the exact same configuration everywhere you're running, the only real answer is to rely on compute resources (basically EC2) and build your own stuff. All the clouds have Linux servers.

I agree with your general point, but switching between Kinesis and Kafka isn't too crazy. SES can be replaced by Mailgun, Postmark, so on and so forth.

The AWS services I use without hesitation are EC2, Route 53, S3, SNS (to mobile devices, not to my applications), and SES. RDS and ElastiCache can be okay for early, pre-scaling situations, and Dynamo is okay for proving out a problem before cranking out a Cassandra cluster (but I wouldn't want to scale on Dynamo, gets real pricey real fast).

I will also heavily recommend Redshift, because it's fantastic for a lot of different offline workflows and it's a lot nicer than maintaining your own. It works well without being tied to AWS for your actual infrastructure.

Because you want to avoid locking your future with one particular service supplier. Particularly one that charges so much (for e.g. bandwidth) as AWS.

Still cheaper than what google charges for bandwidth.

Might be. It's still 10-20x the market price.

Pricing still looks pretty far out there.

Their CPU has a passmark score of 16k[1] and they ask $1300 USD/mo for it.

Hetzner will rent you one with passmark 10k[2] for $60/mo...

[1] http://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2680+v...

[2] http://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-4770+%...

That assumes that there's only one chip in the server (unlikely with 36 vcpus listed for C4), that you're happy with using a desktop chip with no ECC for Hetzner, and your pricing doesn't consider reserved instance pricing for C4 which costs much less than $1300.

The ECC version costs $81/mo at Hetzner.

3 years reserved c4.8xlarge costs $501/mo at EC2.

If you can commit for 3 years, why would you pay a 6x markup for "elasticity"? (11x if you commit for only 1 year...)

Does that server also provide 36 cores, 60GB of RAM, 10Gbps network and the fastest Intel CPU available anywhere? You're comparing a single low-end chip vs a server with multiple cutting edge chips. Everyone knows you pay more for the fastest chips. If budget is the only criterion then there are many less expensive instances.

AWS reserved instances are buying hours. If you can commit to running a particular instance type for ~720 hours every month for a year or 3, then it makes sense to go RI. You don't have to use for one machine running 24x7, you can burst with loads - 2 machines 12x7, etc etc.

Though notably more than the quoted $50, for €69, you get a server chip and ECC.

AWS is expensive for some definitions of expensive, water is wet.

That is a crazy difference though if you can take advantage of it.

I would love to leave my colo for AWS but the numbers just don't work.

you forgot the bandwidth price

Is the Enhanced Networking on a C4 instance a Virtual Function (ixgbev driver) or a full Physical Function (ixgbe driver)?

The docs say that it is a VF but I wonder since they are offering the full 10G of bandwidth.

We could relocate our Snabb Lab to AWS if we got PFs. https://github.com/SnabbCo/snabbswitch/wiki/Snabb-Lab

It looks like it still loads the ixgbevf driver:

  # ethtool -i eth0
  driver: ixgbevf

What does "optimized specifically for EC2" means?

Here's the chip in question (Intel Xeon E5-2666):

http://ark.intel.com/products/81706

I don't think anyone but Amazon has this specific model. I don't see it available for sale anywhere.

Interesting trend. If you could spec out the whole thing including cooling for use in a data center, you could probably do some really damn fast many-core chips that wouldn't be likely to sell much in the desktop and a la carte server markets.

Never mind, this must be it: http://www.theregister.co.uk/2015/01/13/amazons_new_ec2_comp...

There was another article that suggested they were manufactured specifically to amazon's standards for things like cooling and power stability. Instead of having to run in a wide range of environments, they can be designed to run in just the narrow range of Amazon's environment.

see http://www.datacenterknowledge.com/archives/2014/11/13/intel...

It's interesting that this new generation has moved away from onboard ssds.

The extra features of EBS SSDs are pretty useful (ie. snapshotting)

But having attached disks was super fast.

I don't care about disk speed nearly as much as I do reliability. EBS combines inconsistency with unreliability. It is the worst of both worlds for building a reliable, redundant infrastructure. I'm pretty bummed.

I presume they will add storage optimized instances with local disks in the future?

I guess so.

Last time we spec-ed up an ElasticSearch cluster the SSD EBS didn't exist yet, so on-board SSDs was an obvious win.

I should revisit it again some time.

SSD instance storage will always be significantly faster than EBS storage, even if SSD-backed. I'm disappointed that Amazon dropped it from their C4 offering.

For Elasticsearch, we use R3 (memory-optimized) instances which still have SSD instance storage. And boy is it ever fast.

I'm considering switching to these from C3 and it would be interesting to see hdparm outputs / etc on these if anyone is going to try them.

Onboard SSDs are better for some use cases... high volume queues come to mind.

C3 SSDs were crazy faster than even the fastest EBS option listed, plus you could raid0 them...

Fortunately for me, I don't need that much disk space and C3's aren't going anywhere for a while.

Ugh, very disappointed to see these are EBS only.

Agreed, and I think this isn't getting enough play. EBS is one of the more failure-prone parts of the AWS stack. I full-stop don't use EBS, anywhere, on any machine, because of the reliability concerns. (Since I am at least sort of competent, I design for redundancy and don't care if I lose a node anyway, so instance stores are fine.)

These machines sound cool, but C3's will do me just fine.

EBS disks are far more durable (and convenient with S3 snapshots).

Besides my database servers, all of my machines are ephemeral[1]. I can lose machines and spin up new ones automatically (and therefore scale elastically).

EBS adds an unnecessary additional point of failure that I want to avoid at all costs.

Having run production systems in EC2 since 2010, I've survived numerous EBS outages as other services have crashed and burned by sticking to this philosophy.

[1]: I don't use EBS for my database servers either. I use a replicated DB that can lose multiple machines simultaneously without loss of data (and is backed up to S3 in case of a catastrophic event).

Specifically which database?

Cassandra.

There's no question that what you sacrifice in performance with EBS, you get back in easy durability. But if you design your persistence layer with replicated instances, you can have the performance of instance storage. You'll need to make backups yourself, so it is a bit of extra work over making EBS snapshots, but IMO the performance gains are well worth it.

Ephemeral give you extremely high IOPS for no additional cost. It's perfect if you don't need a ton of disk space. A great example is Graphite. Very high IOPS requirement but (usually) a low disk space requirement (assuming you configure relatively low retention) The c3.2xlarge boxes are great for that.

The equivalent on EBS on the other hand will cost you an extra $250/mo or so in addition to your server cost. And you will cap out at 4000 IOPS.

EBS disks are more durable, sure. When you can reach them. You should be designing for failure, and EBS fails not infrequently relative to the rest of AWS. The problem is, when it fails, your instances go with it.

Individual machines' durability is almost irrelevant. Your service's durability is very relevant. This is the most critical consideration in almost all cases in the cloud. EBS makes service durability much harder.

Should we expext R4 (ram optimized) or M4 (balanced) instances soon or is the high cost of DDR4 going to delay things?

I am about to reserve some R3 instances so maybe that will speed it up :-)

Reply on news.ycombinator.com