The term "refactoring" is commonly used in a way which includes changing of public interfaces. Random article, which even cites a book, agrees: https://thoughtbot.com/blog/lets-not-misuse-refactoring
If you're developing a library, then refactoring shouldn't change public interfaces. If you're developing an application and you own all the code paths to the code, then refactoring could change public interfaces, as the external behavior here would be the UI.
If you literally agree with the "refactoring shouldn't change public interfaces" then we need a new word for "code improvement which doesn't change external behavior, which can mean UI", which is the more commonly needed term.
And then perhaps we could agree that "code improvement often changes public interfaces" and how this relates to unit tests.