I can't find a link at the moment, but I recall a paper showing even a very granular clock will suffice for Spectre exploits, albeit with lower bandwidth. Also, something else would need to be done about multithreading, as an application could always just spin up another thread counting as fast as it can to make a poor man's timer.