How much memory bandwidth do large Amazon instances offer? | Hacker News Reader

How much memory bandwidth do large Amazon instances offer?

lemire.me

79 pointsingve43 comments

M1 Pro Macbook 32GB:

Xeon E-2288G (aka i9-9900K) 128GB DDR4-2666:

> You can double this amount of bandwidth with NUMA-aware code, but it will require further engineering.

If you have non uniform memory, you may be shooting yourself in the foot. Author used a dual socket system. If you are not careful, and your code is not NUMA aware, you may end up with less memory bandwidth for your application, than a single-socket would have. Can be avoided by prefixing `numactl --cpubind=0` to your command

Indeed, it's about 50% faster when bound to a CPU: https://imgur.com/a/zk2QjL5, https://gist.github.com/Q726kbXuN/706b51b54d948de2c463893865...

For reference, this is an M1 Pro - the first run is consistently half speed so I’m guessing that’s some kind of scheduler adjustment:

This is why M series chips "feels" faster

No it's not.

Not many codes are bandwidth bound, but you can't say without knowing the use case.

Another factor is mac's have many more memory channels on the m1/2/3 pro and m1/2/3 max. So they can have many more memory transactions, where on most PCs and x86-64 laptops you get 2 cache misses per memory latency (on the order of 70ns), where macs will get many more. So under various workloads all the cores spend much less time waiting on memory latency.

>You can double this amount of bandwidth with NUMA-aware code, but it will require further engineering.

So is this benchmark really useful? Seems like it's engineered enough that it probably isn't relevant for most use cases (like running java, go or python services), and not engineered enough so it's not relevant for performant systems like databases. (That said NUMA simply isn't that hard, even with Java which is also NUMA aware).

Yes, you always have to know your problem well enough to make sure that you pick a benchmark which is measuring your bottleneck. Most people don’t have this kind of perfect linear read activity but some do, and I think that if I did I’d probably be playing with the thread counts to see whether high-core machines were bottlenecking at some point where diminishing returns kicked in making it more cost-effective to get a second system rather than one really big one.

Raspberry pi5 8Gb, with data volume set to 4Gb

E5-1620 v2 @ 3.70GHz, 32Gb

RaspberryPi 4 4GB with data volume set to 1GB (armhf)

RasperryPi CM4 8GB with data volume set to 4GB (aarch64)

Do you have more information on the results obtained? any information on how to enable NUMA-aware code ?

The r6i.metal instance says up to 50 Gbps on Network bandwidth: https://aws.amazon.com/ec2/instance-types/r6i/

50 Gbps is only ~6 GB/s, so you are unlikely to saturate memory bandwidth via network traffic alone

Typically there are NUMA-aware memory allocators. I don't believe glibc is but mimalloc and the non gperftools tcmalloc should be. The Linux kernel is NUMA-aware and will try to avoid shifting work to a different NUMA node but it may depending on what else is happening - it's a generic algorithm trying to balance CPU utilization, latency, and memory bandwidth. If you're using a higher level language like Go or Java then there may be more involved. Java has a NUMA aware allocator while Go's design requires it to also have a NUMA aware scheduler which I don't believe it does.

You could go extreme and start pinning your threads to specific CPUs to tune the code manually instead of relying on the kernel if you know memory bandwidth is extremely important and you won't have much CPU contention to worry about in terms of getting work scheduled in a timely manner.

All that being said, you typically also need to design your application from the ground up to be NUMA aware to take full advantage so that you can set up your allocations to happen on the right zone & whatnot.

I changed data_volume to 8GB and ran on my desktop (Amd Ryzen5 7600X + 2x8GB DDR5-4800 running @ 5000 Mhz)(Win11) I get around 55 GB/s

  PS E:\> .\bandwidth.exe  
  1 54.0  
  2 56.7  
  3 56.1  
  4 55.3  
  5 54.6  
  6 54.3  
  7 53.9  
  8 53.1  
  9 53.4  
  10 53.1  
  11 52.3  
  12 52.8

55 is kinda low for DDR5 I guess..

On S23 Ultra (Snapdragon 8 Gen 2+, 12GB LPDDR5X @4200MHz)

  u0_a339@localhost ~> ./bandwidth
  1 34.4 GB/s
  2 38.7 GB/s
  3 42.0 GB/s
  4 42.3 GB/s
  5 42.4 GB/s
  6 40.2 GB/s
  7 41.0 GB/s
  8 40.3 GB/s

On a NanoPi-R6C (RockChip RK3588s @2.4Ghz + 8GB LPDDR4X)

  pi@nanopi ~> ./bandwidth
  1 22.1 GB/s
  2 26.2 GB/s
  3 27.5 GB/s
  4 27.7 GB/s
  5 27.1 GB/s
  6 27.1 GB/s
  7 27.0 GB/s
  8 26.9 GB/s

Surprising that mobile is almost 80% of desktop in this benchmark.

Running on an 2xARM Cavium ThunderX2 CN9980 with 1TB of ram and 256 threads (Simula Ex3): ~120.

Running on an 2xAMD EPYC 7763 64-Core Processor, with 256GB of ram and 256 threads (EuroHPC LUMI standard CPU node): ~38.

Running on my Macbook Pro M1 16GB on battery power, but with 8GB of data to not swap as I have other apps running: ~59.

Note that you can get a significant speedup by using "restrict" on the pointer in the "sum()" function to tell the compiler that there is no mutation or aliasing happening during the call's lifetime.

On Godbolt, with a size of 100MB (due to limits), this improves the speed using LLVM 17 (https://godbolt.org/z/6dW1h8aev) from:

  1 15.1 
  2 15.9

To (https://godbolt.org/z/sh3489Mxv)

  1 12.8 
  2 13.7

Isn't higher better?

yes. The result is max bandwidth in GB/s https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/...

That __restrict isn't useful to the compiler in this function, since it can already infer that there is no mutation from the access patterns: it's a leaf function with only non-atomic reads and no writes. (It's mostly only useful when you interleave reading from one pointer with writing to another with the same type, or when you call an opaque function before reading from a pointer.) Indeed, if you look at the generated assembly, the two are identical. The timing differences are likely random variation.

7950x3d w/ 128GB at stock timings (~3200MT/s?). Showed a high base but no increase with threads, need to investigate what is happening.

For 2 x 3200 MHz, 51.2 GB/s is the theoretical limit. I guess your single core is good enough to use all bandwidth. How many RAM sticks do you have?

4 x 32GB for now. I need to investigate manual OC as EXPO doesn't work with all 4 slots populated. Another option is to try the 192GB 4x48GB Corsair kits at 5200.

7950x with 32GB @ 6000MT/s:

7950x with 4x32GB @ 3800MHz here. Getting similar results except my single thread performance is the worst (around 40), otherwise it looks almost identical. I assume my poor single thread perf is due to having capped max CPU core clock at 4.5Ghz. Getting 4 sticks to boot was a pain, haven't really bothered to change anything once I finally got it running stable.

Ryzen 9 7900 2x48GB AMD recommends DDR5-6000, so I underclocked DDR5-6400 to DDR5-6000. The 96GB ram was under $300.

$ ./a.out

GCP VM, 40GB:

  Model name:            Intel(R) Xeon(R) CPU @ 2.20GHz
    CPU family:          6
    Model:               79
    Thread(s) per core:  2
    Core(s) per socket:  8


    1 11.4 
    2 21.3 
    3 30.8 
    4 39.5 
    5 48.2 
    6 56.8 
    7 64.3 
    8 71.0 
    9 61.2 
    10 64.1 
    11 67.7 
    12 72.4 
    13 74.3 
    14 78.3 
    15 81.8 
    16 84.9

Interesting stuff, although I'll admit when I read the headline I took it to mean the literal `large` instance types, which are actually quite small as AWS lingo goes. Whereas the author means actual LARGE instances, ie, an entire box with no hypervisor layer.

You could also probably just ask your TAM for this info.

It'd be interesting to compare the results.

As a former AWS employee — answer will vary wildly!

Testing some Hetzner Cloud instances

16-core ARM CAX41

root@ubuntu-32gb-fsn1-1:~# ./a.out

1 17.8

2 35.6

3 49.9

4 64.9

5 79.0

6 92.1

7 101.7

8 110.2

9 117.6

10 124.0

11 130.4

12 136.2

13 140.9

14 143.8

15 149.2

16 152.9

16 Core AMD shared CPX51

root@ubuntu-32gb-fsn1-1:~# ./a.out

1 18.9

2 37.5

3 54.3

4 65.4

5 77.2

6 92.8

7 100.8

8 92.1

9 95.1

10 105.7

11 93.4

12 100.9

13 89.9

14 97.0

15 99.1

16 107.8

16 core AMD dedicated CCX43

root@ubuntu-64gb-fsn1-1:~# ./a.out

1 36.3

2 71.6

3 56.8

4 50.8

5 63.4

6 57.2

7 53.4

8 50.9

9 55.2

10 61.3

11 64.4

12 65.3

13 66.5

14 68.8

15 69.2

16 64.4

Increasing the data volume for CCX43 to 48GB increases the bandwidth using 16 cores to 75.1. Memory bandwidth seems to scale pretty well with the number of cores on ARM. Interesting that the bandwidth for the shared AMD system is that much higher than the dedicated system

Running this code on an "on-prem" AMD EPYC 9454P with 12 x 64 GB DIMMs:

And then it settles around there.

What are the units? MB/s, GB/s

looks like GB/s

So around 3x of what AWS has? I guess this is more about what motherboard they have, and not an on-prem vs cloud difference.

Number of DIMM is important, I know the standard BOM at my company has way more memory that we use just to have more bandwidth.

Bandwidth increases nearly linearly with adding dimms until you get to 1 dimm per memory channel. After that there's very small improvements related to more open pages. More open pages doesn't help with pure bandwidth benchmarks though.

1.5 times greater speed comes from 768-bit vs. 512-bit width of the memory interface.

Another 1.5 times greater speed comes from DDR5-4800 vs. DDR4-3200.

The rest may be from virtualization and other overheads.

For the curious like me, here's the script running on consumer gaming Intel hardware (i9 14900K, 2x 48GB DDR5 6600MHz CL32 XMP, Windows 11):

  1 27.6
  2 50.5
  3 62.5
  4 68.3
  5 75.6
  6 82.6
  7 87.4
  8 90.7
  9 93.0
  10 94.3
  ...repeating

I expected a bit more, my guess would have been around 120 GB/s. I've been playing with LLMs and this hardware is about as fast as memory gets on consumer Intel without overclocking.

Screenshot showing usage: https://i.imgur.com/okdqgG9.png

I changed data_volume to 8GB and ran on my 4 year old laptop (Amd Ryzen 4800H + 2 x 8GB 3200MHz DDR4 running NixOS with Linux 6.1.71) I get around 36 GB/s

I expected 51.2 GB/s (2 x 3200 x 1e6 Transfers/s of 8 bytes each)

Either because it is a laptop CPU or (less likely) because it is an older Zen 2 might prevent it to reach the DRAM throughput.

Running the test on a desktop Zen 3 (5900X) with slower ECC DDR4-2666, i.e. with a maximum throughput of 42.7 GB/s, provides for 2 or more threads a throughput around 39 GB/s, with a maximum of 39.5 GB/s at 4 threads.

Reply on news.ycombinator.com