1. In this application, floating point drift affect matrices far more than quats.
Drifts in quats results in a drifted value of rotation + uniform scaling, but will never introduce deformation.
Drifts in matrices may result in total mutilation of your coordinates.
The overdetermined nature of matrices with respect to orthogonal transforms means that you lose information about which values constitute the authoritative state, whereas it is by definition impossible for a quat not to be orthogonal even when perturbed with significant error.
As an analogy, think of matrices as retained-mode GUI while quats are immediate-mode.
2. Axis angle is just the logarithm of [unit]quats (non-unit quat adds an additional scalar to the axis-angle components).
If you want to compose rotations sequentially, you'll still need to take the exponential of axis-angle to turn it into quats.
You can author initial state in axis angles as an authoritative declaration of what you meant for the orientation to be, but composing them still invariably requires you to un-logarithm them back to quats, hence what I said about "intermediate representation"
3. I said compile to a matrix at the very end when transforming the final vertices, entirely sidestepping the problem of repeated operations since you're only "baking" it for the final transformation onto vertices.