Giving the underprivileged tools to work on themselves is a huge win for society, regardless of the fidelity of all interactions. There are a lot of bad therapists in the world, and not all of them are going to give good advice all the time. But, bad therapists can still provide some information that may be useful to the person learning. Why would we expect perfection from interacting with an LLM instead of a bad therapist? Why be
overly concerned about the possibility of introducing "bad data" in such a session? It could be that bad data then gives way to an opportunity for a leap of logic for the one using the tool. Without critical thinking present, it is unlikely anyone is going to learn to be adaptive over the long term.
Obviously, the use case matters. I'm not saying to give such a tool to someone who is mentally ill, or incapable of critical thought.
In general, I think expecting perfection from LLMs is unreasonable. If someone wants to learn to code, the second they have any interaction with an LLM about coding is the moment they begin to learn. I'm not sure about y'all, but a lot of times when I'm working a hard coding problem, I find myself going through confusion, frustration, interest, and then aggressiveness (anger+interest) to get through the challenge. I frequently tell people who are starting out coding that coding is one of those things that you have to get used "being confused about" to over time. You have to get used to creating problems for yourself, then coping with the feelings that come, and pushing through to a solution.
I do think you raise a profound point about ability to evaluate the fidelity. I would argue that some people will eventually develop critical thought to apply to the problem, but if we instead see a "digital tragedy of the commons" happening, things could go sideways pretty quickly.