The moral argument aside, I don't think the 'safety' argument holds either. For example, an artifact might be 'safest' if it's enclosed in a concrete box at the center of a mountain, but that would be of no use to anyone if it can't be seen. Artifacts like these belong to the people who created them, are most relevant to those people and their descendants, and therefore it is to them that they should be most accessible. For the average person in Nigeria, these artifacts might as well not exist if they're in the British Museum or elsewhere, since they will never be able to see them.
Finally, if the British Museum truly have been good stewards of these artifacts, their rightful owners might decide to loan some of them back, but that is wholly their determination and their choice to make.