Deep Quaternion Networks (2017) [pdf]
arxiv.org
arxiv.org
But quaternions are themselves generalized by Geometric Algebra. And there is a plenty of information about use of GA in the field of neural computing: https://arxiv.org/pdf/1305.5663.pdf (page 3). For example, universal approximation theorem for GA is presented at https://www.informatik.uni-kiel.de/inf/Sommer/doc/Dissertati...
I think that fine article is a step back.
And it is better tooling what I am trying to advertise here.
> Theorem 6.4 ([2]) Complex FCMLPs having (6.9) as activation function are only universal approximators in L∞ for the class of analytic functions, but not for the class of complex continuous functions.
> ... the complex numbers (C0,1) are a subalgebra of the quaternions (C0,2). Hence the quaternionic logistic function is also unbounded. Neither could it give rise to universal approximation (w.r.t. L∞) since this does not hold for the complex case. One may argue that such things become more and more less important when proceeding to higher dimensional algebras since less and less components are affected. This is somehow true, but it hardly justify the efforts.
> ... Summarising all of the above the case of Complex FCMLPs looks settled down in a negative way. ... Hence Complex FCMLPs remain not very promising.
Unless I'm misreading, it seems already known that you _can_ use complex numbers (or quaternions) in neural networks...but you don't really gain anything from doing it.
Even if it was, like, years of studying. I’m just curious how deep this rabbit hole is.
So to get started with reading this paper you just need to learn about deep learning, and then also the very basics of quaternions, which would be taught in, for example, a first course on abstract algebra.
Edit: In an effort to find more applied information I put down my math books and picked up the information theoretic ones. You can find more information about the use of quaternions in the two volume Handbook of Digital Signal Processing and Salomon's Data Compression. More generally, when quaternions aren't explicitly referred to it's helpful to look up the coverage of complex rotations, especially with respect to the Discrete Fourier Transform.
For a discussion of rotations with quaternions in the context of animation, this is a reasonably short paper: http://www.cs.cmu.edu/~kiranb/animation/p245-shoemake.pdf.
I am from Dublin, where quaternions were invented, so they get mentioned a lot by mathematicians and physicists here, maybe getting a higher billing than they do elsewhere. Computer graphics is obviously a place to go for introductions also, but it is typically going to be a more applied and less rigorous treatment.
It's worth observing that what distinguishes complex numbers from 2-vectors like (x,y) is that there's a multiplication rule that corresponds to rotation around the origin. Similarly with quaternions. But you can also just use them as glorified vectors of 2 or 4 elements.
See this https://www.technologyreview.com/s/610278/why-even-a-moths-b... for a recent interesting finding on dimensionality.
Lots of things in math are similar. Simon Altmann's Icons and Symmetries makes a case that using representations with insufficient symmetry impeded our learning of the laws of magnetism.
The algebra of real numbers is simply less structured than the complex numbers. One of the key properties of the complex numbers is that they naturally have both a magnitude and a phase. This lets them capture phenomena that have a notion of superposition and interference.
As you correctly pointed out, you can simulate a complex number with two real numbers. The key is to exploit the particular geometric and algebraic properties of the complexes. One example in neural networks is the phenomenon of synchronization, where the outputs of neurons depending on the presence of a particular stimulus all have the same phase. This can be exploited for applications such as object segmentation.
So the widest possible view of this line of research is that putting more algebraic structure on your parameters can improve the behavior of your learning algorithms. My extremely hot take on how far this can go is a full fledged integration of harmonic analysis and representation theory into the theory of deep learning.
There aren't really any introductions, just research papers. If you have some understanding of real valued neural networks you'll probably be able to work your way through the literature.
I'm coming from a signal processing background, so thinking in terms of magnitude and phase is comfortable to me. Does synchronization, in the sense you're describing, really happen in deep learning (ANN) systems? I'd love a link or reference.
[0]: https://www.haroldserrano.com/blog/best-books-to-develop-a-g...
[1]: https://www.amazon.com/Quaternions-Computer-Graphics-John-Vi...
Take out the associativity and you've taken out the "deep" in "deep learning".
Edit: On further reflection, the non-commutativity of neural networks is also a crucial component of machine learning. Without it, a neural network can't make a decision at one layer that depends on its decisions at a previous level!
If
- your network structure is fixed, and
- you always evaluate the matrix in a given order, and
- you are careful/smart about how you train the weights (Ok, I haven't thought through the ramifications here...)
then I'm not sure you care much about either commutativity or associativity. Maybe a lack of associativity makes backprop impossible, and maybe commutativity makes "Google deep dream" impossible (no idea), but I don't quite agree with the "composability" objection to a lack of associativity and I don't understand the objection to a lack of commutativity sorry.
Toy evaluate the network in two orders - forward (use) and backward (training).
The non-associativity of the octonions is fundamental to their structure, not something to be worked around. In particular, there's no way to consider an octonion-valued network as comprising several layers plugged in serial.
But they're pretty weird; very few of the rules you're used to for "numbers" apply. Ditto for the further constructions.
https://en.wikipedia.org/wiki/Cayley%E2%80%93Dickson_constru...
It isn't explicitly stated there, but the statement I recall from algebraic topology is that the octonions are the last normes division algebra.
Complex numbers - lose self-conjugate identity, but satisfies the fundamental theorem of algebra (and can represent 2D points or vectors)
Quaternions - lose commutativity (but can represent 3D rotation, which isn’t commutative)
Octonions - lose associativity, except for each of aab and abb
Sedenions - lose associativity of aab and abb
John Baez’s “This Week's Finds in Mathematical Physics (Week 59)” http://math.ucr.edu/home/baez/week59.html concludes with a letter by Toby Bartels explaining why. An excerpt:
I will prove below that the 2^n onions are a division algebra only if the 2^(n-1) onions are associative. So, the question becomes: why aren't the octonions associative? Well, I've found a proof that 2^n onions are associative only if 2^(n-1) onions are commutative. So, why aren't the quaternions commutative? Again, I have a proof that 2^n onions are commutative only if 2^(n-1) onions equal their own conjugates. So, why don't the complex numbers equal their own conjugates? I have a proof that 2^n onions do equal their own conjugates, but it works only if the 2^(n-1) onions are of characteristic 2. The real numbers are not of characteristic 2, so the complex numbers don't equal their own conjugates, so the quaternions aren't commutative, so the octonions aren't associative, so the hexadecanions aren't a division algebra.
Awww, I was kinda hoping there'd be ununun-ions.
However, octonions are the obvious next step here: if you look at Appendix Figure 1, of "Deep Complex Networks" [1] , the authors authors used (Real + Complex), and Figure 1 of our paper[2] with quaternions uses (Real + Complex + Complex + Complex)!
[1] https://arxiv.org/pdf/1705.09792.pdf [2] https://arxiv.org/pdf/1712.04604.pdf
Starting with real n-space, one can form the Clifford algebra, which essentially gives a method of multiplying vectors which "knows" something about the length and angle of vectors. The even subalgebra of the Clifford algebra gives a very convenient way of encoding rotations on real n-space. Furthermore, the Clifford algebra is always associative, and works for any n.
If you apply this construction for n=1, 2, 3, you get back the real numbers, complex numbers, and quaternions respectively. If you apply this for n=4, you get back an 8-dimensional associative algebra encoding rotations in 4-space.