A pattern that has saved my bacon a number of times: When writing the "permanent delete" code, only issue the delete when you have proven that there is a backup.

i.e.:

- Create a backup of the data you want to delete and store it somewhere

- Read everything in the backup, and delete everything you find in the backup from prod

- Have the backup expire after a certain time threshold (e.g. s3 has a nice TTL setting for backup buckets)

Even if you delete the wrong thing, the backups are always immediately accessible in the near-term