"Refactoring" is next on my list, but I started with "UML Distilled", and am currently working on "Design Patterns", partly in preparation for "Refactoring" (and "UML Distilled" was partly in preparation for "Design Patterns").
Suggestion : Invert the order. :)
Read Refactoring first than read Design Patterns(GoF).
There's a great book called Refactoring to Patterns which I would suggest after GoF (and just overviewing the Refactoring book)