How do you know where the key is, and how are you guaranteed to be able to read enough of it before the "shifting sands" that is timing unpredictability and general noise in the system make you read something else?
That's what really irritates me about all these side-channels that have been found ever since the first Spectre/Meltdown --- they all demonstrate something flashy like "we read a key/password/secret/something important from memory in a few minutes" while conveniently ignoring to mention the countless hours spent aligning everything just right so they could show off the one "magic trick". In a lot of the cases there's barely even any control over where it reads.
Every time something like this comes up, I feel compelled to post some bytes from somewhere random in the memory of a random process on my machine to show just how much I care; here you go:
E8 7F 00 00 00 A1 64 30 40 00 89 45 D8 8D 45 D8
C4 20 00 00-CE 20 00 00-D8 20 00 00-E2 20 00 00
F8 20 00 00-00 21 00 00-0E 21 00 00-16 21 00 00
26 21 00 00-36 21 00 00-42 21 00 00-56 21 00 00
66 21 00 00-76 21 00 00-84 21 00 00-96 21 00 00
AA 21 00 00-00 00 00 00-FF FF FF FF-B4 16 40 00
C8 16 40 00-7C 20 00 00-00 00 00 00-00 00 00 00
EC 20 00 00-00 20 00 00-00 00 00 00-00 00 00 00
Of course if you are being targeted then the concern may go up, but I still think that attackers would have lower-hanging-fruit than this, seeing as setting up one of these reads to get one secret would itself require such intimate knowledge of your machine's configuration and state that they probably already know what they want.