I wonder how much of the cost is coming from the cache misses vs the more frequent indirections/ILP drop?
For example, I wonder what this test looks like if you don't randomize the chunks but instead just have the chunks in work order? If you still see the perf hit, that suggests the cost is not from the cache misses but rather the overhead of needing to switch chunks more often.