Comment by benrbray | Hacker News Reader

Comment by benrbray | Hacker News Reader

Parent Full thread

I think it conceptually helps a lot to distinguish between row and column vectors, even if they are coalesce in any production code you might write.

(0a) An (nxm) matrix A represents a linear transformation f(x)=Ax from R^m -> R^n

(0b) A linear transformation f(x) = Ax is its own derivative, f'(x) = Ax

(3) The derivative of a function f : R^m -> R^n is a linear transformation with the same "type" R^m -> R^n as the original function. This means it is an (nxm) matrix.

The other two rules can be derived from these rules:

(1) The derivative of a function f : R^m -> R is a (1xm) matrix, aka a row vector.

(2) The derivative of a function f : R^n -> R is an (nx1) matrix, aka a column vector.

This perspective is helpful because, aided by the chain rule, we can easily derive all the other matrix algebra rules on the fly. Distinguishing row / column vectors helps catch conceptual type errors in places where it's inappropriate to add vectors and covectors together.

In (2) I think you meant to write "f : R -> R^n"

You're right, thanks!

> (0b) A linear transformation f(x) = Ax is its own derivative, f'(x) = Ax

f'(x) is just A , not Ax,

f'(x) != f(x)

(I vouched for this comment since it exposes a common misconception people have about derivatives, and I'd rather reply than have it buried)

It's unfortunate that the connection between matrices and linear transformations, and thereby between derivatives and linear transformations, is not adequately emphasized in the math courses that most people take.

The derivative (or differential, or total derivative) of a function f : R^m -> R^n at a point x ∈ R^m is defined, formally, to be a linear transformation A : R^n -> R^m such that

       || f(x+h) - (f(x) + A(h)) ||
  lim  ------------------------ = 0
  h->0          ||h||

Where ||.|| indicates a vector norm that measures the magnitude of its argument. Unpacking a bit, notice that

> h is a vector in R^m

> f(x+h) is the exact value of f at x+h

> f(x)+A(h) is a linear approximation to f(x+h), using x as the base point and A as the linear approximator

So this just a formal way of saying that the dervative A should be a good linear approximation to f.

Every linear transformation has a corresponding matrix, and every matrix corresponds to a linear transformation (with respect to a choice of basis). So, when we say "the derivative of f at x is the matrix A", we really mean that "the derivative of f at x is the linear transformation represented by the matrix A".

EDIT: In response to the parent poster, who has dug their heels in by posting a now-dead rant in reply to this post, I encourage you to refer to a copy of Rudin, "Principles of Mathematical Analysis", for a rigorous treatment of derivatives. See also: representation theory and group actions.

Reply on news.ycombinator.com