Hacker News Reader

Comment by protocolture | Hacker News Reader

>General matrix-vector multiplication (GeMV)

Ok, so my math isnt great.

When I was studying Quaternions during my 3d math class (That I failed the first time, like I said, not a math guy) they briefly covered the history of matrix calculation in graphics development.

My understanding is that Quaternions became popular because they are almost as accurate as matrices but much less complex computationally.

Has anyone tried building an LLM using Quats instead of matrices?

Or are the optimisations with Quaternions more useful in realtime?

View on HN

A matrix is a representation of a linear function (e.g. a function that plays nice with + and scalar multiplication). A specific subset can be used to describe rotations in 3D space. Quaternions can (arguably) do this better. But quaternions cannot be used to describe any linear function. So I do not think this makes sense for LLMs.

> But quaternions cannot be used to describe any linear function

Does this mean all functions that can be described by quaternions are non-linear, or does it mean that quaternions can describe some linear functions such as the ones associated with rotations in 3D space but there are linear function they cannot describe?

Quaternions (when viewed as vectors) are not linear functions, but the arguments to linear functions. You can add them: (a + bi + cj + dk) + (a' + b'i + c'j + d'k) = (a + a') + (b + b')i + (c + c')j + (d + d')k, and multiply them by a scalar: lambda * (a + bi + cj + dk)= (lambda * a) + (lambda * b)i + (lambda * c)j + (lambda * d)k. An example of a linear function on quaternions is the zero function. After all, zero(q + q') = 0 = 0 + 0 = zero(q) + zero(q'), and zero(lambda * q) = 0 = lambda * 0 = lambda * zero(q).

Matrices and quaternions take different approaches to describing rotations: a matrix sees a rotation as a linear function, and quaternions see rotations as a group (confusingly represented with matrices, this field is called representation theory if you want to know more).

So the answer to your question: there are linear functions that quaternions cannot describe. And quaternions can only describe a very specific class of linear functions (with some rather complicated maths behind them).

Quaternions only have four fixed dimensions. For neural networks you need many, many more dimensions.

I think you are mixing things. Quaternions are in the same category as complex numbers. They can be represented as matrices, and there are probably nice uses of matrices where the element is a quaternion (such as QDNNs) instead of a real number. My experience is that in massive architectures such as LLMs, simpler forms are more successful unless there is a true benefit to representing things with more elaborate types of scalars (such as in physics, or 3d graphics).

My understanding was that the main benefit of quaternions in computer graphics was representing rotations in a way that doesn't result in gimble lock.

And beyond that, for those rotations, a quaternion doesn't scale nearly as well as you add dimensions. Complex numbers are a complex representation of two space, quaternions are a complex representation of three space, and to go to four space you have octonions, which have eight elements.

Quaternions have four dimensions.

Yes, but quaternions of unit length are a representation of the rotation group in 3D space ( https://en.wikipedia.org/wiki/Representation_theory_of_SU(2)... ), which is how they are used for rotations.

The original question was: can quaternions be used in place of matrices to perform LLMs tasks, and the answer is: quaternions are 4 dimensions, with the implied meaning that matrices can cover different dimentionalities, which are needed for LLMs (and neural network in general).

Yes, if you have essentially 4d objects and you disable 1 dimension by requiring unit length, you end up with something that is effectively 3d.

Of course, that the 3d thing you end up with represents rotations in 3d space is extremely neat; and not something all 3d things do.

Axis-angle also doesn't have gimbal lock - the main advantage of quaternions is that actually performing rotations with them only involves addition and multiplication, no trigonometry at all. The same is true for using proper rotation matrices, but those use a lot more memory. Plus, you can actually lerp between quaternions (more generally - they compose). That doesn't work with matrices (I think)

Axis angle has gimble lock when composed.

Reply on news.ycombinator.com