I don't think I've ever seen unit tests that didn't need to be changed fundamentally to deal with any sort of refactoring more complex than extracting a function.

Unit tests are supposed to work with the internals of a module and not just its boundaries. Since refactoring normally tries to significantly change the internals without changing the boundary, it usually also results in a rewrite of the unit tests. Integration tests are the ones that usually help me sleep better after a refactor.