It's extremely annoying, but usually the attacker will switch to another domain after a few hours or days. The mail notifications can be delayed by a few days, depending on the individual mail server's settings, so you should see a slowdown of the incoming bounce messages after a while before it will stop eventually.