>Boiling it all down, when prompted, these models just regurgitate a similar sentence to what is observed in the training data for loosely that same input, using some glorified curve fitting

This is not that much different than what you do.

What criteria would you use to determine if something understands the meaning of a word/phrase/concept that isn't a string of definitions and metaphors? And at what level is sufficient?

Attempting to prove that something "understands the meaning" is a fruitless task with no quantifiable criteria - much like proving something is "conscious."