Simply require that a SIM can only be swapped if it is disconnected from the mobile network for 48 hours. And if it isn't disconnected, the original SIM will be called/texted to ask if they really want the SIM swap to happen.
If someone has both devices in hand, there isn't even need for a delay. The only time you need a delay is when the original device is missing. In that case, sending a message to that SIM and having a mandatory delay (ideally, customisable by the customer) seems reasonable.
> You have requested a replacement sim card. To proceed with the replacement now, reply "Yes". To keep this sim card, reply "No". If you do not reply, a replacement will be mailed to your billing address: 54 Wolverton Gardens in 7 days, and this sim will be deactivated.
An attacker now has to overcome the time delay, and the fact that the replacement sim card must be mailed to the billing address. For those people who have an outdated billing address and lose the sim card, require the sim to be offline for 7 days, or demonstrate access to an email address or credit card on the account.