Paper lists 3 configs, one with 128GB and two with 256GB.
I'm guessing most of the perf comes from the gpu memory size.
While that's likely true, I generally find that it's quite rare that enough attention is paid to how information is moved between disk, RAM, CPU and GPU. And paying close attention to that can be extremely helpful. Taking the RAM up to 11 can eliminate a lot of the art to it, which is a good thing.
A machine in this class can easily have a terabyte of RAM. Add a couple Optane DC sticks and you have enormous storage at exceedingly high bandwidth.