Another angle: the PCA is given by computing the SVD (a more general analog of eigenvalue/eigenvector decomposition) of a whitened representation of the data. Some idiosyncrasies of PCA them become obvious: we can't determine if a computed result is actually a sought result or its reflection/negative, because the SVD is only unique up to sign variance.

This is also closer to it's actual implementation: while it's true that you do technically need the eigenbasis of the covariance matrix, you should not actually form the covariance matrix to get there...