The Pantheon is a temple dedicated to all the ancient Roman gods. It cannot be both that and the Church of St Mary and the Martyrs. When the Church took it over, it ceased to be the Pantheon.
There's no doubt the building is the same, and we can argue about how 'true to form' it is. But that's the point. When it stopped being what it originally was, its 'form' is now a goal to never deviate from.
Whereas, the various depictions of the martyrs present within the Church of St Mary and the Martyrs, for example, has no such ideal yet. The Church can (and does) shift that around. However, if it were to stop being the church of st Mary and the Martyrs, then it would have an ideal form that any restorer may want to restore it towards.