This makes no sense. Your model typically isn't asked to make predictions on "old data" because it was trained on it. And understanding why it made certain prediction isn't any different on "new" vs old data.
You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this.
Or, like in the article, you can make use of introspective models that can tell you what dimensions were fitted. This can more easily show you where overfitting is occurring.
You can see the weights of the model and follow the decision network, which allows you to explain why something has occurred in a much clearer fashion.
An introspective model gives you a much clearer picture of why certain choices were made, rather than the fuzzy picture you get running statistical analysis against a black-box'd algorithm.