If you don't tell them, yes.

As many websites do not convey the language properly they try to fix that using some heuristics, but especially for small texts (like messages) that can easily fail hard leading to completely wrong pronunciation.

Mail for example tried to fix that by allowing you to annotate the language for UTF-8 embedded in mail headers and the content. Ironically while that mechanism works for display names it doesn't work for email addresses itself. And I'm not sure any mail program uses it.

The same Han character can have wildly different pronunciations even in a single language (being a logogram, they represent a word and not a sound). KS X 1001, the primary Korean character set, even duplicated same characters according to their readings so that they can be almost [1] correctly converted back to Hangul. In practice they didn't work well though, and Unicode assigns all but one duplicate characters into the compatibility region.

[1] These readings didn't take account for systematic variations like the initial sound law (두음법칙, for example 이 vs. 리 at the beginning of words).

You can't split CJK sentences into individual letters, inspect them one by one and decipher its exact meaning. If you present Chinese writing to a Japanese speaker, they'd only see complete gibberish consisting of letters they may or may not recognize. It also works the other way around too.

On top of that, Kanjis aren't the only type of letters Japanese folks use. They also use Hiraganas and Katakanas, which are phonetic symbols and totally unrecognizable to non-Japanese speakers.