Have you explored using a formal verification tool to prove out the safety/liveness properties of your (or the end user's) autonomy specifications? Could you model the system in something like Spin/NuSMV/PRISM and then ensure that certain properties hold (useful states are reachable and dangerous ones are not)?