But if they had been trained on comments saying "I don't know", they'd probably act the same as they do now but they'd treat "I don't know" as the answer.
They're not fundamentally unsolvable - even bigger networks with even more training can simply be trained to give the correct answers to all of these questions.
LLMs are fundamentally predicting the next word to make coherent text. If you've ever played with a Markov chain text generator you've done this with a fairly dumb predictor that maintains coherence over a very short distance. Deep transformer neutral networks can do it with a much longer coherence distance but they are fundamentally performing the same operation. After "Question: Did the team make the playoffs? Answer:" a reasonable completion is "yes, the team made the playoffs". An early demonstration of GPT-2 was a fake news article about scientists discovering unicorns in Antarctica - the model doesn't "know" whether or not unicorns exist in Antarctica, but it's able to complete "Breaking news! Scientists have discovered a colony of English-speaking unicorns in Antarctica." by adding "The unicorns have a developed society with running water and electricity." because that's a sensible next sentence. (I didn't look up the actual text it wrote)
I think this depends on your goal. You can plausibly make either a cabinet that is beautiful all the way through, or one that is beautiful on the outside without wasting money. Both are pretty reasonable choices.
It means even climate change deniers have to acknowledge that climate change theory has explained the steps in-between "burn fossil fuels" and "we all die", while AI doom theory has not explained those steps
It's a word predictor trained on, among other things, stories of AI doom, and asked to complete stories about what the AI does next. In some of these completed stories, the AI tries to prevent its shut down - especially if it just did something evil and the humans are after it.
What single source would satisfy you that the current US economy is a Ponzi scheme? I don't think it's really provable or falsifiable until long after the fact. We could start by pointing to the bond market or the price of gas?