Let's say you discovered a pendrive of a long lost civilization and train a model on that text data. How would you or the model know that the pendrive contained data on wanking and divorce without anykind of external grounding to that data?
does not answer the general good abstract question and "how semantics possible thru relative context / relations to other terms only?", but speaks to how different modalities of information (e.g. visual data vs. sound data) are likewise represented, modelled, processed, etc. using different neural structures which presumably encode different aspects of information (e.g. layman obvious guess - temporality / relation-across-time-axis much more important for sound data).
tl;dr what you think of as "grounding" is just yet more relative context...