This was right as generative AI was coming out and back when it would struggle with this type of analysis.
The first analysis was done the old-fashioned way – you had to understand what you were looking at, you had to know what to do with the data, and had to be willing to work through uncertainty. The assignment: take this raw data and make recommendations to the channel owner on what to do (left open for interpretation).
Results? - 30% of submissions were terrible. - 60% were good, though not really complete. - 10% were amazing, producing creative recommendations, and better than I expected.
The semester after that I did something I never did before: I assigned the same thing. The same data and instructions, but this time with the requirement to use AI in the analysis.
Results? - Much of the worst work decreased. Only 5% of submissions were terrible. - 90%+ were good. - But at most 5% were amazing. AI usage had mostly eliminated the worst results (30% in the first attempt), but it had also hurt the top output (the 10%), with results condensed toward the center. In some cases this might be the outcome you want, but in other situations you might want to take some terrible results so that you can also benefit from the excellent ones.
Or be disciplined enough not to let AI completely think for you.