This is not necessarily a trivial fact, but I wouldn't call it incredible. It says a net trained with gradient descent can fit the input perfectly if its much larger than the input.

But this says nothing about generalization, i.e. performance on test set -- which is what we really want.