Nice. Definitely true that evaluating agents behavior is by far the toughest part of building them. Also most eval cases are added without thought and not maintained when agent behaviour updates. Interesting approach.
Would love your thoughts on this: https://fabraix.com/blog/adversarial-cost-to-exploit