Actually, the view that Mathematics was "discovered" and not "invented" is quite popular among Mathematicians. There is an area of research called the Foundations of Mathematics, which investigates this question.

It's untrue that Mathematics does not undergo any refactoring -- in fact, it does: even in a field as new as Algebraic Geometry, the "classical" textbooks which build up the subject using ideals and varieties are very different from the modern ones which start from Scheme Theory.

Moreover, Category Theory shows that there is a beautiful underlying structure in all fields that can be extracted out into a new field. For example, the concept of adjunctions existed much before Category Theory was invented -- CT came along and showed that similar structures exist in Algebraic Topology, Algebraic Geometry, Differential Geometry, and other fields.