You may or may not want to exclude the effects of caching, it depends on what your goal is. Famously, for IO benchmarking you often want to flush the cache before starting a benchmark. But for algorithms you may be very interested in how the algorithm interacts with the various caches.