Is Google’s New AI as Smart as a Human?

I can't watch this at work, but as I'm subbed to Two Minute Papers, I'll definitely be watching it later. I see from the video description it's about Minerva, though, an AI I read about the other day. In my understanding, Minerva is basically a version of PaLM specifically fine-tuned to solve math and physics problems, and some of the common mistakes it makes are similar to human mistakes. To me, this raises the question: what do we mean when we say an AI is "smart" or "as smart as a human"?

If we just mean "is it as good at physics and math problems as an average high school student", then probably, yeah. But the fine-tuning means it's likely less generalizable than the base PaLM model would be, and I'd argue any AI that's truly "as smart as a human" would need to be generalizable. Because there are so many ways to measure intelligence, we'd need to account for them all to do a direct comparison to human "intelligence".

So while this is certainly impressive, to me it's less impressive than more general LLMs including the base PaLM model itself.