It is just a function. A next token prediction function. Garbage-in/Garbage-out is as true as ever.
Coding is solved. Specification and translating requirements into specifications is an unbounded domain and unsolvable by definition.
What LLMs change is that now you develop the specification through iterative implementation and testing.
You start with a weak specification which produces slop, then you progressively build the specification with different approaches to automated testing. The tests are the specification. Ultimately, what you ship is the tests. They are the only thing that proves the functionality of the program. An LLM can produce code that passes any test suite you give it. If the result is bad, the problem is not the code, the problem is the specification.