Why the assumption that only the implementation needs to change, especially when significant refactoring is involved? The interfaces themselves may need to change when striving for a better system design. This is also part of refactoring.
What you are describing is not refactoring: "In computer programming and software design, code refactoring is the process of restructuring existing computer code—changing the factoring—without changing its external behavior."[0] The interface is an integral part of external behavior.
With that said, what is "internal" vs. "external" behavior is… fluid. A large-scale refactoring might involve changes to internal components with their own interfaces and tests. The tests for the component being refactored should not be affected, but in the course of refactoring the larger component you might make more extensive changes—not just refactoring—to the smaller pieces making up its internal architecture. For that you would need to design new interfaces & requirements for the internal components, write new tests, and then iterate on the implementation until the tests pass, just as with any other TDD process. In the meantime you have the unmodified tests for the larger component to verify that your new internal architecture still satisfies those higher-level requirements.
Sure, in with bigger refactorings, some unit tests probably change. That's just a (small) part of everyday work, as I do it.