LLM also have no idea what it is capable of. This feels like difference to humans. Having some understanding of the problem also means knowing or "feeling" the limits of that understanding.
2. LLMs regularly tell me if what I'm asking for is possible or not. I'm not saying they're always correct, but they seem to have at least some sense of what's in the realm of possibility.
2. That is not my experience. Almost half of the time LLM gives wrong answer without any warning. It is up to me to check correctness. Even if I follow up it often continues to give wrong answers.
2. What are you arguing about? I didn't say they're always correct obviously.