Imagine a person is born blind, and from birth is kept in a room and never allowed outside. They are provided with food and a bed, along with a library of audiobooks to listen to whenever they want. From those audiobooks, they learn all about the world around them: about trees, and grass, and wind, and water, and buildings, and birthdays (they've never celebrated), and fireworks, and puppies, etc. Would you say that, because this person has never experienced these things other than via language, they don't -- and can't possibly -- understand anything about the world? I would argue they certainly do understand, and an LLM is doing the same thing.

In a less extreme, hopefully more relatable, example: think back to when you were in school. Everything you ever read in a textbook or on a whiteboard/blackboard or in a homework packet. Are you telling me that either (a) you didn't learn any of it, or (b) you personally experienced everything you read about in order to learn it? Because if there's something you understood by reading about it, despite having no other sensory experience of it, then you learned by finding patterns in the language -- you must have, for it was the only data available to you.

Yes, multi-modal learning has the potential to enhance that learning greatly (hence why multi-modal models are all the rage this year), but humans do in fact learn through language alone sometimes, and we still call that understanding.