The effect of ChatGPT on students' learning performance, perception,and thinking

I'm still reading through the article, but as a teacher, I'm highly skeptical of the results of this meta study. My own observations do not support that ChatGPT et al improve learning performance and especially thinking.

As you can imagine, discussing AI in education is a huge topic. But seeing a study like this that just says "Yep, it's all good." I'm stupidly skeptical of this sort of meta-study. I very much believe the jury is out, and especially in terms of long term performance. I don't care if my students are able to use ChatGPT to "do better" in my classes if the results are short-lived.

A short review of my experiences; my bias, I am very much skeptical of the generative LLM's on the whole. I do fear it will become a crutch rather than a teaching tool. But I also use them, and I think it's amazing technology. I see it as a tool that will be used. And as it's a tool, like all tools, it can be used for good or bad.

For every amazing anecdote I have of a student using ChatGPT to do something amazing or using it as a personal tutor (and I have seen this and it's awesome.), I also have anecdotes about students relying on nothing but ChatGPT output, taking it as gospel, and imo, losing their research problem solving capabilities. I've taken to calling it the "Death of Research." "Why bother reading or looking for anything, ChatGPT will just give it to me."

I will say that I deal with STEM education, work mostly at a Community College in a trade, and as such, I'm exposed to a very different cohort of students that tend to be less computer literate, and are different in many ways than your traditional students. But I also work with high school students, who are very computer literate, and ironically, these are the students I see the problems with.

I'm not trying to say "AI Bad." But I sure don't buy the argument "AI Good."

This study I feel is misleading at least at a headline. They found 6621 studies and narrowed it down to 51 studies to analyze. This is somewhat fine as meta-studies do need to narrow from the initial scope, but this is more aggressive than I have seen in the past. I haven't had time to view all of the studies, but from what I can tell most of the studies are somewhat specific. That is, they are asking "If we give students ChatGPT in specific situation (specialized prompts, supervised environments, etc) then can students succeed?" I worry that the headline is quite misleading in this instance. The overall study seems to ask "if we can align the stars would ChatGPT be helpful?" This is a useful question to ask, but certainly not equivalent to "is ChatGPT helping students learn right now?"

tl;dr:

>The overall effects of ChatGPT on enhancing learning performance, improving learning perception, and promoting higher-order thinking were calculated as g = 0.867, g = 0.456, and g = 0.457, respectively, indicating that ChatGPT has a large positive effect on learning performance and a medium positive effect on learning perception and higher-order thinking.

----

I would think that as schooling modernizes, there ought'a be separate components (one involving digital devices, the other not). I'm old enough to remember teachers admonishing "You won't always have access to a calculator," so the current tech already overwhelms...