I disagree heavily.
Refactoring tests is as important as refactoring "production" code. Following the four elements of simple design (http://www.jbrains.ca/permalink/the-four-elements-of-simple-...) you can use test refactorings to find better names and detect need for abstractions.
Tests with lots of duplication, for example in the test setup, get hard to maintain and developers tend to abandon them more easily (my own experience).