It's unclear that you really need to convert to matrices before sending off to the GPU. [1] finds that quaternions are faster; I find that questionable (does it scale?), but it shows that the two are comparable.

1: https://tech.metail.com/performance-quaternions-gpu/