In some ways this is similar to those game demos people get the LLMs to build. When you say "build me a cool cyberpunk FPS" you get the FPS it can build, not the FPS the author wanted, or the FPS that is desired by players. It looks impressive but that doesn't make it a good game, or the game anybody actually asked for. It's demo porn.

In the same way if you tell an LLM to go and find an unsolved cipher it can solve, of course it finds the one it can solve out of the set of all possible ciphers. Of course it finds one that uses a one time pad that is public and referenced nearby in the text.

It's the same trick used by those people who film themselves throwing a basketball backwards into the hoop. You do it enough times and don't show the misses. You pick the best one to show. It makes it look like you're a basketball genius when you aren't.

It is of course, still a cool trick. Those videos are fun to watch, and so is an LLM solving a cipher. It is absolutely incredible to live in the timeline where you can tell a computer in plain language to go and find a puzzle on the internet and solve it, and it does exactly that. It's truly a mind boggling miracle.

The first principle is that we must not fool ourself, and ourselves are the easiest people to fool. (Ht Feynman)