Hacker News Reader

Comment by phreeza | Hacker News Reader

If they are only orthogonal if they share no bits that are set to one, only one vector, the complement, will be orthogonal, no?

Edit: this is wrong as respondents point out. Clearly I shouldn't be commenting before having my first coffee.

View on HN

I don't think so. For n=3 you can have 000, 001, 010, 100. All 4 (n+1) are pairwise orthogonal. However, I don't think js8 is correct as it looks like in 2^n you can't have more than n+1 mutually orthogonal vectors, as if any vector has 1 in some place, no other vector can have 1 in the same place.

Hmm, I think one correction: is (0,0,0) actually a vector? I think that, by definition, an n-dimentional space can have at most n vectors which are all orthogonal to one another.

It's not correct to call them orthogonal because I don't think the definition is a dot product. But that aside, yes, orthogonal basis can only have as much elements as dimensions. The article also mentions that, and then introduces "quasi-orthogonality", which means dot product is not zero but very small. On bitstrings, it would correspond to overlap on only small number of bits. I should have been clearer in my offhand remark. :-)

Your initial statement is still wrong, that you can include a lot of information in a small number of bits. If you have a small number of bits, the overlap will be staggering. Now, that may be ok, but not ok, if you want to present orthogonal concepts (or even quasi-orthogonal).

Also, why do you believe dot product cannot be trusted?

What I meant was similar to the article. If I have bit vectors of length 1000 bits (that will be the embedding). Let's say that every concept I want to model corresponds to a set choice of 10 bits being all set to 1, and the remaining bits 0. (These vectors are quasiorthogonal.) Then I can easily store a sentence (bitvector) about 50 concepts, while there is relatively small chance of overlap, i.e. all of the concepts are decodable from that bitstring.

But it's quite similar to what the top comment is saying about spherical codes. I think my comment is also about using coding theory to represent concepts.

Other than that, I don't have any issue with dot product over bitvectors - it's just not very useful for the above.

By the original definition, they can share bits that are set to zero and still be orthogonal. Think of the bits as basis vectors – if they have none in common, they are orthogonal.

For example, 1010 and 0101 are orthogonal, but 1010 and 0011 are not (share the 3rd bit). Though calling them orthogonal is not quite right.

Why not? The 1010 and 0101 are orthogonal.

Reply on news.ycombinator.com