You can get 48GB dimms now for 96GB from just two sockets.

But yes, there's definitely a memory speed limitation there.