Bear in mind that "any two high-dimensional vectors are almost always orthogonal".
I have mostly a laypersons understanding of this idea but I would assume that it would be false to say that they are typically _entirely_ orthogonal?
If you're picking a random point on the (idealized) Earth, the probability of it being exactly on the equator is zero, unless you're willing to add some tolerance for "close enough" in order to give the line some width. Whether that tolerance is +/- one degree of arc, or one mile, or one inch, or one angstrom, you're technically including vectors that aren't perfectly orthogonal to the pole as "successes". That idea does generalize into higher dimensions; the only part that doesn't is the shape of the rest of the sphere (the spinning-top image is actually quite handy).
That said, for sparse high dimensional datasets, which aren't proper vector spaces, the probability of being truly orthogonal can be quite high - e.g. if half your vectors have totally disjoint support from the other half then the probability is at least 50-50.
Note that ML/LLM practioners use "approximate orthogonality" anyway.