I'd add an additional safeguard. When retrying failed jobs, make sure to have exponential backoff. Numerous queueing systems have this feature built in.
The issue comes in when it doesn't know a job actually failed, as was the issue in this case. Since the process was `kill -9`'d delayed job wasn't aware the job failed and attempted to run it again as though it was the first attempt.