Hacker News Reader

> 3a. As a result, we have to run R4 instances for some of our web tier - despite not needing the memory - because they have the highest network allocation. To make matters worse, AWS won't tell you the network allocation. You don't know until you start dropping packets.

Ugh, I hate that. They also have hidden limits on the number of incoming tcp connections you can have.

Worse still, we spent weeks on the phone with AWS insisting there was no throttling - just to find out there was throttling. And that would have been discoverable if a single engineer had looked at CloudWatch.

Do you have Enterprise Support? If so, this is a very surprising result.

Just Business Support, we're not that large of a shop.

> They also have hidden limits on the number of incoming tcp connections you can have.

Do you have proof of this?

No, I had an experiment I was tried to run on EC2 in 2013, and ran into this. It was very clear though. Established connections would plateau, and then no more tcp syns would arrive to the ec2 host, unless a connection was closed. At the time the limit was 5000 connections on the micro instance -- from my notes, we got an allocation for hi1.4xlarge and I know we hit the limit there too, but I don't recall what the limit was. This was a very simple TCP proxy, using Linux kernel ipmasq; this uses significantly less memory than HAProxy, although with a lot less features.

We had a very excited account rep because of where I work, but he was only barely able to confirm the limits were there, he wasn't able to get them raised or removed, nor could he tell us the limits by machine type.

There's a thread from 18 months ago on the aws forums [1], where an aws rep more or less confirms, but again provides no information

[1] https://forums.aws.amazon.com/thread.jspa?threadID=231806

The "Connection Tracking" portion of http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-net... provides some insight here and describes a method to avoid connection tracking in EC2's firewall.

Thanks, this looks like what I needed to know!

> I've been hoping for their release since their announcement in November 2016 (and they were already late for Skylake at that point)

I don't know what you mean by 'late'. Skylake Xeons were delayed and have only been released recently, at least partially. You may be able to get them from vendors like Dell but you still can't go out and buy one anywhere that I'm aware of.

Does the updated documentation at http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitorin... explain the CloudWatch metrics?

That's more helpful. That's a lot of special cases - and thought - to put into each time we look at a C5-attached volume as opposed to any other. It also does not include any mention of Average Read Latency and Average Write Latency, which appears to be incorrect in all dimensions (average, min, max, sum).

We've found that CloudWatch is useful for some system and I/O related metrics (i.e., the from the EBS SAN's point of view) but less useful for others (i.e., metrics from the VM's point of view). It's worthwhile to install a monitoring agent on the VM that can collect I/O latency and other statistics there too. We use Datadog, but there are lots of options out there (collectd etc.).

This is a topic that we're continuing to iterate on between EBS, CloudWatch, and the AWS console team for metrics. When a newly introduced behavior makes step function changes in graphs displayed in the AWS console it doesn't meet the principle of least astonishment.

We're also continuing to investigate the reported latency in the console. Because this is a derived metric from VolumeTotal{Read,Write}Time and Volume{Read,Write}Ops, there may be a miscalculation happening due to the change in dimensions.

Not sure if this is related, but when I was getting EFS working last year I noticed that CloudWatch graphs were often complete nonsense due to EFS not logging zeroes for idle filesystems but CloudWatch treating this as "missing data" rather than "implied zeroes".

#2 is expected to work. Can you reach out to me directly (joemag@) and I will make sure we take a look.

There's an open support ticket, but I've reached out as well.

So I wouldn't say C5s don't work at all with NLB instance target groups - I'm running one right now.

I wouldn't be surprised if you're hitting an edge case with the tighter integration* between NLB's and instance associations. If you haven't already, please do reach out to support.

* from the docs "If you specify targets using an instance ID, the source IP addresses of the clients are preserved and provided to your applications. If you specify targets by IP address, the source IP addresses are the private IP addresses of the load balancer nodes."

It appears the specific issue is with an NLB in one subnet referencing a C5 in another subnet. It's useful to put the targets in a private subnet and the NLB in a public subnet, so this breaks usage for us.

You have given me a whole other perspective to an aws customer. I spend a lot on aws (70k/m +) but id be happy if i had core2duo cpus!

What sort of work do you have that requires the latest generation ? Or more why do you want the latest ?

Id expect aws to always be behind the ball - are they the right platform for you?

Gce is interesting, im moving half of my infra over there - but again its not really about their hardware offerings. Its more about being multi cloud/redundant

Our particular use case involves a few processes that are heavily serial, while yet memory-intensive; the fastest possible processor would be a boon to us (for instance, if someone would guarantee an overclocked Xeon, we'd take it in a heartbeat for almost any price).

AWS is indeed behind the curve more often than not, but they also have some amazing hosted products. That was a bigger deal in 2014 than it is now (Kubernetes will eat the world) but it's still a reliable offering, and reliability is our most important criteria.

Agreed re: being multi-cloud in the end. It's the only responsible choice above a certain scale.