I agree that a lot of progress in math comes from refactorings and novel concepts that generalize neatly. My point is that those breakthroughs don't happen through refactoring in a vacuum, they happen because the refactoring is undertaken with a specific motivation. It often ends up having more general applications than the original motivation, but doing it without a specific motivation doesn't usually yield progress.
It's the same as with software refactoring. If you refactor without a sense of what you want to get out of the refactor, how do you know whether you're refactoring the right things?
(Also just my opinion)