Garbage in, garbage out. The language model based it’s response on a vector input, where the values of the vector represent words in different positions. If you point the vector towards garbage, you’ll get garbage.

As a side note, if AI is a system that can self improve then most corporations are AI but language models are not.