If you haven’t seen it, I’d highly suggest looking at some of the recent research on in-context learning and reasoning in LLMs.

It is becoming increasingly clear that there is a phase-change in behaviour when these models get large enough, such that they can solve new tasks outside of the training distribution.

See work by Hattie Zhou or Laura Ruiz for example.

It is clear to those following the research or using these models that they are not just copy-pasting… (Even if you can cherry pick examples were the LLM recalls highly occurring dataset items like fast square root or whatever).