Yeah, you would think so. But it really hasn't improved all that much over the years. (Of course, that's partially because it's hard to do!).

Roughly speaking, when I designed with DRAM in 1983, the access time from RAS was 150 ns for the most popular parts. Now, with DDR4, you're talking CAS latency of about 10 ns, an improvement of 15x.

In the same timeframe CPU clock rate increased from about 10 MHz to about 3000 MHz, an improvement of 300x.

I grok your comment about perhaps having two levels of DRAM, but the whole memory hierarchy thing is already crazy. Most computers already have 3 levels of cache ahead of the DRAM.