That shouldn't be happening, can you confirm? It should flush out the old ones based on the current DNS information. The whole point is to automatically update the alerts based on the latest DNS information so you don't need to manage that yourself.
If you want I can re-add it and ping you if it happens again. (where can I ping you?)
I also had a support ticket around that time about this but never got an answer, guess it got lost.