That's right Colin. I completely agree, however if you break unit tests during a refactor, yet there is no functional change - bin the tests as long as you have end to end tests higher up. I should add that any new code you add should have unit tests around. E.g the new code added as per the refactor if needs be.
With good unit tests you can indeed refactor without fear, but we should not take the stance of never breaking any unit tests. Providing there is no functional change, break away! Such stance relies on end to end tests/pairing/some other process to ensure you are not breaking valid behaviour though.