if quaternions, why not octonions?
However, octonions are the obvious next step here: if you look at Appendix Figure 1, of "Deep Complex Networks" [1] , the authors authors used (Real + Complex), and Figure 1 of our paper[2] with quaternions uses (Real + Complex + Complex + Complex)!
[1] https://arxiv.org/pdf/1705.09792.pdf [2] https://arxiv.org/pdf/1712.04604.pdf
Starting with real n-space, one can form the Clifford algebra, which essentially gives a method of multiplying vectors which "knows" something about the length and angle of vectors. The even subalgebra of the Clifford algebra gives a very convenient way of encoding rotations on real n-space. Furthermore, the Clifford algebra is always associative, and works for any n.
If you apply this construction for n=1, 2, 3, you get back the real numbers, complex numbers, and quaternions respectively. If you apply this for n=4, you get back an 8-dimensional associative algebra encoding rotations in 4-space.
Take out the associativity and you've taken out the "deep" in "deep learning".
Edit: On further reflection, the non-commutativity of neural networks is also a crucial component of machine learning. Without it, a neural network can't make a decision at one layer that depends on its decisions at a previous level!
If
- your network structure is fixed, and
- you always evaluate the matrix in a given order, and
- you are careful/smart about how you train the weights (Ok, I haven't thought through the ramifications here...)
then I'm not sure you care much about either commutativity or associativity. Maybe a lack of associativity makes backprop impossible, and maybe commutativity makes "Google deep dream" impossible (no idea), but I don't quite agree with the "composability" objection to a lack of associativity and I don't understand the objection to a lack of commutativity sorry.
Toy evaluate the network in two orders - forward (use) and backward (training).
The non-associativity of the octonions is fundamental to their structure, not something to be worked around. In particular, there's no way to consider an octonion-valued network as comprising several layers plugged in serial.
But they're pretty weird; very few of the rules you're used to for "numbers" apply. Ditto for the further constructions.
https://en.wikipedia.org/wiki/Cayley%E2%80%93Dickson_constru...
It isn't explicitly stated there, but the statement I recall from algebraic topology is that the octonions are the last normes division algebra.
Complex numbers - lose self-conjugate identity, but satisfies the fundamental theorem of algebra (and can represent 2D points or vectors)
Quaternions - lose commutativity (but can represent 3D rotation, which isn’t commutative)
Octonions - lose associativity, except for each of aab and abb
Sedenions - lose associativity of aab and abb
John Baez’s “This Week's Finds in Mathematical Physics (Week 59)” http://math.ucr.edu/home/baez/week59.html concludes with a letter by Toby Bartels explaining why. An excerpt:
I will prove below that the 2^n onions are a division algebra only if the 2^(n-1) onions are associative. So, the question becomes: why aren't the octonions associative? Well, I've found a proof that 2^n onions are associative only if 2^(n-1) onions are commutative. So, why aren't the quaternions commutative? Again, I have a proof that 2^n onions are commutative only if 2^(n-1) onions equal their own conjugates. So, why don't the complex numbers equal their own conjugates? I have a proof that 2^n onions do equal their own conjugates, but it works only if the 2^(n-1) onions are of characteristic 2. The real numbers are not of characteristic 2, so the complex numbers don't equal their own conjugates, so the quaternions aren't commutative, so the octonions aren't associative, so the hexadecanions aren't a division algebra.
Awww, I was kinda hoping there'd be ununun-ions.