This is something AMD is already doing for EPYC, and they've already used HBM2 in their GPUs.
So I'm surprised they haven't released any CPU models with crazy huge L4 caches using a few GB of HBM2.
Then again, Intel made a laptop CPU with a huge 128MB cache and their comment was that it didn't make that big of a difference. I believe the performance boost was less than 5% for going from 64MB to 128MB.
960MiB is prodigiously large for such a microscopic chip, but if it "only" gain 3-5 times latency reduction over external DRAM, it's still very far from a proper L3 implementation, and far behind L2.
Make DRAM work on 1Ghz+, and then you will see miracles. Imagine a fully synchronous on-die DRAM that can sit just behind L1, or even be connected to load registers directly.
The problem is that effective frequencies for memory round-trip haven't got up much since nineties. If you work with 100% cache misses, your mem will still be working at effective frequency of around 100 to 200Mhz
I think you mean 20MHz. Current DRAM is abysmal at random access.
Even after bytes arrive to DRAM controller analog side, a lot of things have to happen before the data gets to the register. This accounts for the further five to tenfold increase in round trip latency
Cache-as-RAM is also a thing, and allows the sort of pinning you described. Pretty specialized use cases though. (https://stackoverflow.com/questions/41775371/what-use-is-the...)