> The susceptibility of these systems to completely failing on weird inputs shows that it's a kind of massive memorization going on here, rather than true understanding.

Isn’t that a truism? “Understanding” is equivalent to AGI. Nobody would argue that we have AGI yet, but the intelligence-sans-understanding is somewhat similar to animal intelligence, which was the precursor to human intelligence. What should scare us is that we know that animal to human was not a difficult step evolutionarily.