It is highly unlikely to just be semantics, because when it comes to visualizing complex objects as a non-aphantasic, the specific details spring forth unbidden as an image first, and then you attach the language, and it isn't the reverse order. I suspect also tests of visual memory of complex pictures should reveal similar things here, because e.g. language and other things don't really seem to be able to represent such details, and neurobiologically, there isn't much plausible other way your brain could be memorizing a complex photo without retaining / having activation patterns similar to the actual viewing (and this is what we see in fMRI studies).
If you are careful at noticing your thoughts (and not aphantasic), you can also notice if you are a highly visual thinker, you might sometimes confuse certain memories of things (spellings, place names, other things) that are similar only visually, but not logically or in any other sense modality representation. This would suggest the representations are genuinely visual in these cases.
But yes, no way to be 100% certain, but the "it is just semantics" case is very weak and can't explain much.