If aphasia is evidence that some of us don't use language to think, then language is nothing more than a programming language.
Whatever programming language using language is irrelevant to people with aphasia.
Regarding language being in conflict with other modes of thinking, there is indeed a "savant" effect of aphasia, where after an aphasia-inducing stroke people suddenly develop amazing abilities in visual art, music, or mathematical thinking. But it is not consistent - most people don't develop these. And it comes with impairment of emotions, memory, etc. So really what the evidence suggests is that some people suppress parts of their brain, and these injuries unlock that potential because the suppression mechanisms break. It's most likely a cultural thing - people act how they are "expected" to act. There is some evidence that females actually have more biological capability for math (in cultures with very high gender equality) but typically you see less performance, so the conclusion is that the culture essentially "programs" in the lower performance. It is probably the same with the supposed "conflict" between thought, action, and language - the culture treats these as distinct modes and inhibits cross-modal thinking like synesthesia.
Both are abstractions that use symbolic representation
Both are designed for human understanding
Both have quirks that make them better or worse at certain kinds of abstraction
comparison != analogy.