This is literally an entire paper about constructing a specific incantation to create an effect. Neither the author nor the LLM maker can completely deconstruct and trace every steps of the process from the input to the output. They only know how to chant a litany, add a request, chant another litany, then look at the output and hope they didn't summon Satan.
Could you explain what you mean here? To the best of my knowledge, there hasn't been much success in successfully explaining how LLMs actually work? Of course we know all the low level mathematical details, we built them. But my understanding is that we don't really know much about the structure of LLM parameters and how they relate to the concepts the model is supposedly learning.
"... even to its creators"
-- me
Sure you can, that's how my brother learned to be a mechanic.
Still, there's a deeper analogy here. Very much in the "enshittification" vein, today's tech companies are about making something cool, getting folks to use it, and then monetizing while gradually turning said cool thing into a bad parody of itself.
-- someone wise