Potentially not, many MTA relays will retry a couple of times before completely giving up.

The sender might also get a delivery failure message from the relay.