Workloads on Arm-based AWS instances
bignacio.github.io
bignacio.github.io
To me, this implies they used a single ec2 instance of each size. However, ec2 instance p99s or so can be impacted by "noisy neighbors", especially on the burstable types which are intentionally oversubscribed.
It's still useful to know if, for example, t4gs are more prone to noisy neighbors, but with only 1 instance as a datapoint, you simply can't tell if it was bad luck or not.
I think this test would be much better with either only dedicated instance types, or by running it with a large n such that an individual unlucky/noisy-neighbor doesn't influence the results overtly.
> It does appear that the Arm-based instances can’t consistently maintain the same performance at high request rates.
I'm unwilling to trust that statement at face value for now given it's been tested against a 't' instance.
EDIT: Removed note about network burst credits in compute and memory optimized instances. I'm not sure if these instances have that.
> A credit specification is only available for T2, T3, and T3a instances. Selecting Unlimited for the credit specification allows applications to burst beyond the baseline for as long as needed at any time.
So that would mean Unlimited is not a setting available for T4g (ARM instance) and therefore may explain inconsistent behavior in the ARM instance.
> T4g instances start in unlimited mode by default, giving users the ability to sustain high CPU performance over any desired time frame while keeping cost as low as possible. For most general purpose workloads, T4g instances in unlimited mode provide ample performance without any additional charges. If the average CPU utilization of a T4g instance is lower than the baseline over a 24-hour period, the hourly instance price automatically covers all interim spikes in usage. In the cases when a T4g instance needs to run at higher CPU utilization for a prolonged period, it can do so for a small additional charge of $0.04 per vCPU-hour. In standard mode, a T4g instance can burst until it uses up all of its earned credits. For more details on T4g credits, please see the EC2 documentation page.
I've just launched a test t3 instance and it didn't pop up anything.
They're using KVM under the hood, which supports things like live migration, interrupts to facilitate memory transfer, and so on. At home I'm doing live migration with less than 300ms of interrupts with high memory pressure. It would make sense for them to move things around, just for things like maintenance and distributing load.
1) More fine-grained bins to help show the shape of the distribution (are there performance cliffs?). Try using vertical lines to denote % cutoffs.
2) Given the wide range between your bins, a log scale might be a good idea instead of raw frequency.
3) Try some other method of visualization. I'm not sure a histogram is useful for what you're trying to convey, at least the way it's being used here.
As it stands, the visual information is so dominated by the 99.5% case that the plots don't help illustrate your tabular data.
Instead of a histogram, a violin plot (especially "split") might be an excellent way to compare the distributions. And that solves the bucket size problem too IMO.
https://seaborn.pydata.org/generated/seaborn.violinplot.html
All of that to say, I think the t3/t4 instance used in this test is a bit problematic for getting a true idea of performance.
[0] - https://depot.dev/
Also 2, t4g instances use Graviton 2 which has, relatively speaking, weaker cores. To get best experience you would need to compare versus Graviton 3 (but these are more expensive, but you can deploy to them in a denser manner).
Such instances share the vcpus, and only get burst of dedicated cpus, then they'll get throttled for performance.
The author should have picked any of the other instances. The bare minimum to make an informed decision should be the c6g.medium or the c7g.medium.
In my experience btw, the c7g family really seems to be closing the gap in single-threaded performance with x86-based instances.
Random numbers:
- an Ampere Altra Max Q80-30 (80 cores) has a 210W TDP
- for the same power draw, you get an AMD EPYC 9334 which is a 32c64t monster
At 2.5x the core count, even if the ampere ones are less powerful, there still is a huge gap. Those are not the CPUs used by AWS but this still gives us a ballpark: there is a substantial performance gain at the same power target, or a substantial power savings at the same performance target.