There exist problems where finding the solution is hard but verifying the solution is easy. example in the general case: prime-factoring numbers.

Are you claiming that for all possible problems posed, verifying the correctness of the output of an LLM is equal or higher difficulty than solving the problem itself? If so, that seems like a claim more arising from emotion than reason.

In the general case, you can have it generate code in a proof based language that "proves" the code is correct against formal specification. Unless you consider math itself to be "bullshit" too.

Within the constraints of technical systems, it's pretty obvious. It's not a topic that you can really bullshit - things work or they don't. It makes mistakes sometimes and what it says won't be consistent with other things it's saying or that you know to be true. If you ask it if it's sure, it often will correct itself. Worst case, you verify with other sources and/or try it out.

It's not really any worse than asking someone knowledgable about a subject face-to-face. They know a lot, but they can be mistaken about things. At some point you will tease out the misunderstanding.

The best way to understand the value of the tool is to try it for yourself.