Why not drop servers in a way described here? http://serverfault.com/questions/249316/how-can-i-remove-bal...
The DNS resolution work that is going into 1.6 releases may lead to a workable approach where you over allocate servers in a backend and then update them to have the right host/port. Last I checked, however, you still couldn't set the port dynamically.