LLMs answer what unfoundly self-convinend people answer when you ask them for diretion on the street. Convincing, not necessarily correct. They are trained on the “data highway“ mostly. A highway full of that type of people, so that behaviour was to be expected.