Yes, of course as I referred to this is something only high-value targets need to worry about, and even then I think it's not that high up on the list of risks. 5 days is just to read the data, and there's a considerable amount of preparatory work involved in setting up this attack --- figuring out
what to read and
where it is, is just as hard if not more so than figuring out
how to read it through Spectre.
The authors of that paper have the massive advantage of knowing exactly the software running on the target system and its environment; something which an attacker in the real world is unlikely to have, unless the attacker already has such familiarity with the system that it seems far easier to exfiltrate data via some other means than trying to find and setup this very slow side-channel. Everything has to be set up just right for this to work. Otherwise you might probably still manage to read something, but it's completely useless.
(High-value private keys in companies are likely to be in HSMs anyway, in which case they're completely inaccessible to attacks like these.)