While you and I know that refactoring is the right thing to do, the customer/manager isn't going to say "by all means, delay completion of the project in pursuit of improvements to parts of the system that only you have to deal with". They don't see/understand the code so naturally they don't care about its state.
[note] Obviously this is a blanket statement but it's been applicable to me and I'm sure others have experienced similar.