Integration tests, however, should not have to change, since they are typically run against the external interface to the software, which shouldn't change if you're just doing a refactoring.
They absolutely do, but at higher levels of abstraction, in terms of interfaces, interactions, constraints, obligations and responsibilities. In fact, unit and integration tests make you think about design in essentially the same way.
Having good integration tests means that you're free to completely re-organize your code, and you can still test that it works as expected.
Given that requirements change over time, even a well thought out code structure will likely need to be refactored at some point, and being able to have a test suite that doesn't have to be re-written during a refactoring is extremely valuable.
I've also seen code bases become unnecessarily complex because so much thought was put into how to unit test it, that not enough thought was put into writing clear, easy to understand code.