When a transaction was rejected by the Bitcoin network that MtGox sent, the MtGox software would detect the rejection and immediately re-credit your account for the attempted transfer amount. MtGox also had an api that allowed you to see the exact contents of the transactions that they sent to the Bitcoin network. This meant that what someone could do is just grab the rejected transaction, fix the malformed portion, modify the transaction id hash and resend the transaction, causing it to go through, but MtGox was unaware of the successful transfer and would re-credit the account. The user could then rinse and repeat over and over.
The interesting twist of this is that it means MtGox knows which user accounts were used to steal coins from them since the malformed transaction could be modified to change the transaction id hash, but the receiving bitcoin address could not be modified without invalidating the transaction.