There are other forces at work.
For instance on my primary workstation after a period of time from cold boot, pretty much everything is sitting comfortably in 32 GB of memory. A lot of time is more due to the
1) sheer volume of crap that has to be shuttled in memory, because memory accesses still aren’t free 2) there’s a surprising amount of repetitive CPU intensive tasks just to get a program started up on merely bookkeeping tasks like linking, conversion, parsing.
In addition to all the built in latencies already mentioned.
To prove this you can setup a system using a RAM disk and notice that responsiveness is often still subpar.