Another interesting point is that from a numerical optimization point of view, the Jacobian transpose method is analogous to gradient descent, while using its (pseudo) inverse is basically the Gauss-Newton algorithm.