What I was trying to say is that sometimes you need to provide additional context and information to those not using the same information model. Maybe it's not compression, but rather you have to include the 700kb of your information model with your message.
The LLMs don't have access to private information other than what you give it. If you give 300 bits of information to your LLM, the information it has is now "all public knowledge + 300 bits", and any text it generates is a subset of that information. By definition, it can't add any more information. If you give those 300 bits directly to me, I (and optionally my LLM if I want) now have "all public knowledge + 300 bits + my private thoughts", which is a superset of what your LLM has. Anything your LLM can infer can be inferred by me and my LLM. All your LLM can do is repackage that information into different text.
My opinion is that I don't get value out of that repackaging. I would rather read your packaging of those 300 bits rather than the LLM's packaging of those 300 bits.
For this current discussion, we could use an information model saying that your LLM has a knowledge base private to just you and it (e.g. local files or other conversations), and say that's separate from the prompt you give it. It sounds like maybe that's the information model you're thinking within.
In that frame of reference, maybe your shared knowledge base has 200 bits, and you type 100 bits into your LLM's prompt. My argument would still be that you're "giving" 300 bits to the LLM, and you should instead give them to me by sharing your knowledge base, or giving me the relevant information in your knowledge base in your own words. The latter option is definitely more work for you, and is the weakest point in my argument, but I'll still hold that preference.
I don't believe that it holds true in context of typical communication between humans. You need to include the additional text to save people time looking things up, make the story flow better, fit in within the expected style of writing for intended audience etc.
To me the problem with LLM writing is not that someone is sending me 700kb of information I can get elsewhere, but that they didn't invest the time to understand what would be useful for me as a reader. I don't need 700kb of generic slop, but I probably need additional 300kb of relevant publicly available info to help me understand the purpose of the new information being sent.
In my opinion, not understanding the reader is exactly why bad writers remain bad writers - LLM doesn't know who the document is written for exactly.