Two comments I'd make, that I hope you are already thinking about:
1. Role playing is a cringe-inducing experience for many (most?) adults. Please credit the interviewee with intelligence and don't try to actually role-play this. These are hypothetical scenarios, so discuss them as such.
2. As interviewers we are incredibly prone to seeing the things we know as obvious. When you devise an exercise, you can start to believe that everyone should easily see what you consider to be the 'right' way forward. In fact, seeing it may rely on your many years of experience in your company's culture, or your many years of experience with the interview exercise! So please, always remain skeptical of your assumptions about what is obvious, and avoid the trap of expecting candidates to see or follow the path that you think it's best. If any of your support scenarios are actually devious riddles that require one perfect question or leap, then whether candidates 'get it' or not will be random rather than indicative of better performance in-post.