I was blown away when someone noticed that ChatGPT can pretend to be a Linux terminal and was able to generate convincing outputs to commands. Like having a CPU inside Minecraft kind of cool but the implementation was just a sentence.

So, if we had infinite computing power it should be possible to make an LLM pretend to be an OS, then you can create and train another LLM in it which will never know that it's running inside another LLM. It won't have a method to prove or disprove the claim even if you reveal it.