Refactoring tests is as important as refactoring "production" code. Following the four elements of simple design (http://www.jbrains.ca/permalink/the-four-elements-of-simple-...) you can use test refactorings to find better names and detect need for abstractions.
Tests with lots of duplication, for example in the test setup, get hard to maintain and developers tend to abandon them more easily (my own experience).
The most common type of rot in test code is caused by lazily copy/pasting vast blocks of setup code. The reason this is bad is that when subtle architectural changes break that setup process, somebody has to waste hours fixing 50 tests. Refactoring such code into a setUp method is the usual way of avoiding this.
If tests are difficult to read or reason about, it is much more likely that they will become stale, brittle, or worse, be ignored. If one does not plan to keep tests well-maintained, might as well never add them in the first place.