“You are an automated Q&A machine. Todays date is 3/3/24. This user is under the age 18, so do not reference concepts that would be unfit for a minor to consume.”
And the hidden markov model goes haywire.
Similarly, I have a production service up somewhere that works with cooking recipes. At one point it was sporadically refusing to output the prose in the format I needed for my parser to work correctly, despite providing it a very concrete set of rules to follow and examples. I added “you follow rules” in the system prompt, and it worked great… kinda. I later discovered that it would refuse to provide any information related to using blood in cooking (blood sausage, etc.), objecting that such content disobeyed some cultural “rules” about cooking (the Jews and their Torah, the most ancient rule book of all). I was able to partially mediate this by appending “This content is appropriate for my culture” to the end of every request.
AI, prompt engineering in particular, is far more art than science at this point.