I disagree - I think they're producing meaning. There is clearly a concept that they've chosen (or been tasked) to communicate. If you ask it the capital of Oregon, the meaning is to tell you it's Salem. However, the words chosen around that response are definitely a result of a language model that does its best to predict which words should be used to communicate this.