If we relax the restriction that the columns of the 3x3 transformation matrix must be unit vectors, we also get the ability to apply scaling and shear transformations. In solid mechanics, this is called the F tensor, and is used to describe deformation of materials [1].

There is also an augmented 4x4 form if translation is needed as well:

  | Q ...  Δx |         | vx |
  | .      Δy |         | vy |
  | .      Δz |   dot   | vz |   = new vector
  | 0 0 0  1  |         | 1  |

where "Q" is the 3 × 3 transformation matrix (e.g., rotation matrix) that is the subject of the OP, and [vx, vy, vz, 1] is the augmented form of a vector "v" = [vx, vy, vz] that is being transformed.

The augmented form is especially useful for transforming voxel indices in an 3D image array to spatial coordinates, such as for MRI or CT image data [2]. It is helpful if the index coordinate system has its origin at zero (zero-indexed arrays), like a normal coordinate system.

Finally, if we have many [x, y, z, 1] data points that each have an old and a new position, we can compute the overall best-fit transformation from the old to new positions with the least-squares solution to

  A x = b

where "A" is the (unknown) augmented matrix representing the transformation, "x" is a 4 × n array storing the points' old augmented-form positions, and "b" is a 4 × n array storing the points' new augmented-form positions. If we allow the points to have arbitrary dimension, without any particular spatial interpretation, we get least squares curve fitting and the foundation of machine learning.

[1] http://homepages.engineering.auckland.ac.nz/~pkel015/SolidMe...

[2] https://nifti.nimh.nih.gov/nifti-1/documentation/nifti1field...