Retries become dangerous when workflows aren't designed
to be idempotent from the beginning.
What helped us was treating every job execution as replayable and attaching a unique operation key instead of relying on execution state alone.
Otherwise retries silently create data inconsistencies that only appear much later.