Not OP, but I have been poking around with LLMs for a year and I'd like to add to the conversation.

In my experience, LLMs are word predictors, and the impacts of that fact are not immediately obvious.

LLMs are capable of "explaining" what code does. What it is doing under the hood is pattern matching: I've seen code that looks like X, with an explanation that looks like Y

LLMs are capable of formatting text. It has seen English written like X, that is reformatted to look like Y

One resounding fact my team has found over and over is that "the things we think are hard for LLMs aren't necessarily hard; the things we think are easy aren't necessarily easy"