It'll be interesting to see how kids will respond to AI hallucinations. Will they just accept it as fact?
The 'critical thinking' part at this point in time leads you to only one conclusion: that the tool is unreliable and dangerous and shouldn't be used. Or, you're teaching kids that they need to cross-reference all 'authoritative sources' which leads to a complete failure of the system - if you give kids a textbook will they have been 'trained' by GPT to not believe a word of any of the claims in it without cross-referencing? How does that enable better education.
If you provide a tool from a position of authority it needs to be a reliable and believable tool; standing in a position of authority and handing out access (to children!) to a lie box and asking those children to 'rely on it' is frankly abusive. That's beyond mixed messaging, that's just some form of weird mental torture.
I just think it's a real issue to give kids free access to a tool that so confidently makes stuff up. I know that textbooks aren't perfect but they're at least audited by a committee of experts, mistakes slip in but there's oversight, there's a procedure to ensure that they're reliable representations of (currently understood) facts, and that when politics enters the textbook production process it (correctly) becomes a bit of a national scandal.
And its enough when such a technology is 80% right. Nobody will verify every answer if it is correct most of the time.
The issue arises when the belief you have is tied to your ego so you only cherry-pick the evidence that will satisfy your ego and ignore/question everything else, but not this.