Invalidating cache lines is not that hard, it happens all the time. Refilling them from DRAM takes time.