kNN with k=1 is a more complex model than a neural network, not a simpler one. If we had more data it would scale and if we reach a point where we have a labeled image for every possible pixel combination in MNIST it would be unbeatable.