While true, it's important to ensure there is adequate test coverage before trying to refactor, in case you miss something.
Also, try to avoid small commits / changes; churn in code should be avoided, especially in kernel code. IIRC the Linux project and a lot of open source projects do not accept 'refactoring' pull requests, among other things for this exact reason.