Isn't it the converse(?)? Words mean whatever the audience understands them as.
A good speaker will define the words as he speaks (still relying on some baseline shared understanding of the most common words, of course; there is only so much time in the day...) so there is no room for confusion, but in absence of that it is expected that the listener will question any words that show an apparent disconnect in meaning, allowing the speaker to clear up what was meant, to ensure both parties can land on the same page.
Even more so when it's an author/reader relationship. The reader is free to interpret the book/article/etc. how they want, and if enough agree, it becomes the consensus.
Where audience understanding is important than you will definitely go out of your way to ensure that definitions are made abundantly clear and that the audience agrees that they understand.
But, in actual practice, most of the time the audience understanding really doesn't matter. Most people speak for the sake of themselves and themselves alone. If the audience doesn't get it, that's their problem. Like here, it means nothing to me if you can't understand what I'm writing.
While the work is being written, the author is the only known audience. In practice, if an author cannot find enough motivation to write something for himself, it simply won't get written. Anyone else who happens to become the audience is gravy, but the work cannot be written for them as they only become known as an audience later.
I think we're quibbling over what "matter" means. If you look at it from this perspective, I understand where you're coming from.
In my opinion it's almost exactly the opposite - once something is written the only thing that matters is the audience.
But that doesn't mean the audience came from the same place. It is very possible, and often happens, that they heard/created an entirely different definition for the same word. The speaker cannot possibly use their definition before even knowing of it.
If you are trying to tell something to someone you should say it so that they someone understand what you want them to understand.
It does not mean that the audience can on the fly redefine your words, it just means than you cannot expect them to magically read your mind