I don’t know how to square this with the fact that I routinely ask GPT-4 for code and get back answers that are in fact correct. You’re making it sound like it’s purely random, like I’m just getting back a string of zhrueksnbd8373&;! or something.
Accuracy rates much, much higher than chance are not accidental. While you're pointing to a valid failure mode of LLMs, you're not using the correct words to describe it which is leading you to say nonsense.