Refactoring can impact your unit interface too, and when that happens you'll have to refactor your tests accordingly. What you don't want while refactoring is being stuck with your old interfaces.
Exactly. Refactoring is the process of changing code without changing behaviour. If that means you change some tests so be it. There is no rule to say that you can't do this, but many devs seem to fear this.