In my experience, you can switch lenders if you are having a difficult time negotiating fees (the escrow is a different party).