> It doesn't mean that these LLMs are about linguistics and grammar specifically.

Correct. But it does mean they're about "language" rather than knowledge. It's meant to read, parse and generate language, no matter if it includes prose, facts, knowledge or whatever.

> Clearly, they aren't as likely to generate "Colorless green ideas sleep furiously" as "It's important to feed your dog".

Clearly, as the training set probably contains much more instances of the latter, than the first example. And if it's more probable to generate, that's the text it'll generate. It doesn't understand why or how, just that it's more likely, so that's the way it's going.