Have you ever tried the Gandalf AI game?[1] It is a game where you have to convince ChatGPT to reveal a secret to you that it was previously instructed to keep from you. In the later levels your approach is used but it does not take much creativity to circumvent it.