There's probably more than one perspective on this. Here's one: a standard way to approach SVD is to start with a matrix A and consider A^TA and AA^T. Since the latter is symmetric (or hermitian if on a complex vector space, where transpose is replaced by adjoint), by the spectral theorem it has real eigenvalues and orthonormal eigenvectors, from which the SVD can be deduced.
But the definition of A^T or A^* depends on a particular choice of basis, and is not a coordinate-invariant concept. If one had an inner product <,>, the adjoint can be defined in a coordinate-independent way using that inner product: A^* is the operator defined by
<Au,v> = <u,A^v>
for all vectors u and v. (One can check that A^ is uniquely defined.) For a different choice of inner product, one gets a different A^.
Note that the operator A^TA comes up naturally when one considers what the action of A does to the length of vectors:
|u|^2 = <u,u>
|Au|^2 = <Au,Au> = <u,A^Au>
Geometrically, the SVD tells us how a linear transformation dilates or contracts space in different directions, e.g., think about what a linear transformation does to a sphere. But all these concepts -- length of vectors, spheres, ellipsoids (images of spheres under linear transformations) -- depend on a choice of inner product.