I actually I fashioned this logic out of the philosophy question of why certain neural firings appear as sound in our brain while others appear as vision? What gives?
does not answer the general good abstract question and "how semantics possible thru relative context / relations to other terms only?", but speaks to how different modalities of information (e.g. visual data vs. sound data) are likewise represented, modelled, processed, etc. using different neural structures which presumably encode different aspects of information (e.g. layman obvious guess - temporality / relation-across-time-axis much more important for sound data).
tl;dr what you think of as "grounding" is just yet more relative context...