The problem with quaternions for optimizations is that they are over-constrained. Meaning the norm of the 4 values must sum to 1 for it to be a valid 3D rotation. So you need to ether penalize a rotation with a norm different from 1 or constantly remap the quaternion back to a valid one in each iteration of the optimization. Neither solutions is very elegant.

A 3 valued so3 lie algebra vector is however always a valid 3D rotation, no matter what the values are. That makes it an ideal parameterization of the optimization problem.

The mapping from so3 lie algebra to SO3 lie group (which could be represented as a rotation matrix or quaternion) is a smooth differentiable function, which allows us to compute the jacobian we need for the gradient in the optimization.