The trouble is counterfactuality, ie. what exactly we're talking about if we say "I went shopping, but I could have not done so."

(Personally as a compatibilist, I assert it's a map-territory error; the counterfactual worlds represent evaluated decisions, not real universes.)