My best guess about this result is mentions of "mirror" often occur around opposites (syntax) in direction words (semantics). Which does sound like a good trick question for these models.
Bubeck, Sébastien, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, et al. “Sparks of Artificial General Intelligence: Early Experiments with GPT-4.” arXiv, March 27, 2023. http://arxiv.org/abs/2303.12712.
or watching Sebastien Bubeck's recent talk he gave describing what GPT-4 can do that previous LLMs couldn't: https://www.youtube.com/watch?v=qbIk7-JPB2c
Geoffrey Hinton recently gave a very interesting interview and he specifically wanted to address the "auto-complete" topic: https://youtu.be/qpoRO378qRY?t=1989 Here's another way that Ilya Sutskever recently described it (comparing GPT 4 to 3): https://youtu.be/ZZ0atq2yYJw?t=1656
I'd also recommend this recent Sam Bowman article that does a goood job reviewing some of the surprising recent developments/properties of the current crop of LLMs that's pretty fascinating:
Bowman, Samuel R. “Eight Things to Know about Large Language Models.” arXiv, April 2, 2023. https://doi.org/10.48550/arXiv.2304.00612.
[1] https://arxiv.org/abs/2210.13382
[2] https://twitter.com/leopoldasch/status/1638848881558704129
[3] https://www.reddit.com/r/naturalism/comments/1236vzf/on_larg...
Q: A glass door has ‘push’ written on it upside down. Should you push or pull it
A: If the word "push" is written on the glass door upside down, it is likely that the sign is intended for people on the other side of the door. Therefore, if you are approaching the door from the side with the sign, you should pull the door instead of pushing it. However, if there are no other signs or indications on the door or its frame, it may be helpful to observe other people using the door or to try both pushing and pulling to determine the correct method of opening the door.