What transaction mode would you need to make this safe? Can you explain further?
My understanding is that in general, if you hit an exception in postgres at all then you can't trust the isolation of the current transaction anymore.
That's what MERGE and speculative insertion (on conflict do update) addresses.
REPEATABLE READ or SERIALIZABLE isolation levels would help with that, as the name suggest repeatable read ensures that the read made by inserts constraint check is repeatable in successive select (or update) statements.
https://www.postgresql.org/docs/current/transaction-iso.html is pretty comprehensive.
Most RDBMSs don't behave this way. They allow you to correct exceptions that occur for anything less than a deadlock without a total rollback, but not Postgres.