I wonder if this could be solved by moving in-progress jobs to a separate table...

Guess you lose job atomicity that way.