This is very much not my area of expertise, but there are a few things that I think could be challenging. Let's assume that a cached resource loads in ~0.1 seconds and an uncached one loads in ~1.0 second:
* Any random delay of up to less than 0.9 seconds is completely pointless since we would know that any resource that loads in 0.5 seconds has to be cached - the 0.4 seconds spent waiting is useless.
* A random delay of up to a second creates ambiguous cases: is 1.1 a cached + maximally delayed response or an uncached but only slightly delayed response? But, with a random delay, we're still going to have pretty different peaks in the expected latency graphs for cached and uncached resources - uncached ones would peak around 0.5 seconds and cached ones would peak around 1.5 seconds.
* A random delay much more than a second would start causing the peaks on our expected latency graphs to be closer together - at a 10 second delay, we'll have a peak in the graph around 5 seconds for cached resources and 6 seconds for cached ones. I'm a bit fuzzy on how many resources we'd have to load to get a statistically significant finger print. I suspect its not all that many - but I could be wrong. However, we're also talking about a pretty significant delay by this point.
We also have the issue of how to handle AJAX requests. You only get one shot to measure the initial load. But, after that you can make an AJAX request to re-load that same resource with caching disabled and measure that. I'm not a statistician, but, I suspect that that information will be pretty helpful in figuring out what was cached vs not cached. Of course, we could add in some random delays here too - but since we can measure this an unlimited number of times, I suspect this would be even easier to defeat.
So, my suspicion is, that these type of random delays are possible to defeat if you have a better grasp of statistics than I do.
But, we could also assume that that is not the case and that they can't be defeated - what does that get us? We've had to add all these random delays in to block the side channel which is going to hurt latency - the very thing we're trying to improve with caching. I also suspect there is a ton of complexity to consider if you want to avoid having all of the same random delays on repeated visits.