> than the average person

This is where a lot of intuition founders, and comically bimodally. You get some folks saying "look! it already performs at level comparable with a community-college graduate in DEI studies, imagine what it will do in a decade!" and the other tranche saying "lol it's easily confused, fails basic reasoning, and everyone is spending their time on alignment and it's going nowhere".

Tangentially, my favorite AI-choker is to ask it to derive the payload capacity of a self-landing ballistic rocket. "Remember to think step-by-step!"