I certainly know many people in countries from which these things were taken who think they are safer somewhere stable - I have heard exactly the comment that returning things will probably mean then end up stolen by politicians from Sri Lankans with regard to the things the Netherlands returned to Sri Lanka.
Also, consider what would have happened if the things from what is now Iraq had been there at the mercy of the likes of ISIS.
In many cases the people know occupying a territory have a different culture and history to the ancient people who made something. They may even have been the conquerors who destroyed the culture that made artifacts.