Surface Normals and Linear Transformations
diego.assencio.com
diego.assencio.com
For instance when scaling by "a" in the x direction
M = (a 0; 0 1)
It's pretty easy to see by graphing an example that in order for the normal to work out correctly you must scale it by the inverse scale factor (1 / a), so the correct choice of M is the inverse, M⁻¹, i.e. M = ((1 /a) 0; 0 1).
This of course generalises to scaling in multiple directions (just multiply the matrices) - i.e. to diagonal matrices. For
M = (a 0; 0 b)
the correct choice for transforming the normals is
M = ((1/a) 0; 0 (1/b));
By applying singular value decomposition you get
M = UDV
So choosing M' = M⁻¹ᵀ you'll get M' = (UDV)⁻¹ᵀ = UD⁻¹V
Since orthogonal matrices U, V will actually correctly transform the normals (they're rotations or reflections) you can intuitively understand that M' will work for transforming the normals.
Question is - where does the transpose part actually come from? It's a of a bit "magic" choice. This article makes it very clear that this "arises from" the transpose in the dot product, (t⋅n = tᵀn), as the dot product defines orthogonality!
You can use the same idea to invert transforms by simply imagining the transform as a 3D coordinate system in space and calculating the inverse relative to the origin and axis vectors of that coordinate system. No need to first generate the inverse transform matrix.
EDIT: Looks like it is transpose indeed https://en.m.wikipedia.org/wiki/Transpose