That might be true for one-hot vectors but it's not true for learned embedding through the lens of attention. That said, I only made to page 3/9 of the paper before the mark-up for the math went over my head.
I can't of course be certain in all cases, but dimensions are typically (past experience, and using knowledge from word2vec experiments from years ago) derivative of higher dimensions. The kernel still operates on the same concept by applying a norm along with whatever weightings to each dim.
Semantic understanding is still not there in my opinion, we might feign it by increasing specificity, but only so much. Largest contributor will likely still the determining factor rather than the series of smaller, more specific dimensions.
I tested this using similar sentences as my original comment and failing in more scenarios than passing. I of course am biased since it may be given I did not select the right dimensions or measures.