How differently is linear regression than PCA? I understand the procedure and methods are completely different, but isn't linear regression also going to give the same solution on these data sets?
Linear regression has a notion of input and output. You want to model how the output depends on the input.
PCA does not. You have a pointcloud and want to find a compressed representation of it.