A model (and a learning algorithm) should be interpretable if we can clearly state the assumptions, and prove that under these assumptions, we get what we state we get.
We can predict the movement of the planets (short term). This is interpretable, because we only assume a model of 3d space, and Newton's laws(1). The rest is mathematics, which gives us elliptic trajectories, etc. The resulting predictions may be wrong, due to wrong assumptions, but the model is still interpretable.
Linear regressions, histograms, decision trees(2) are all interpretable. We know exactly what they do, under proper assumptions. Yes, these models can be sometimes manipulated. But we know this precisely because we know exactly what they do. Whether we should use these models, or statistics in general, in courts etc, is an ethical decision, and perhaps practical decision, similar to whether we should use the death penalty. Has nothing to do with interpretability.
With CNNs, at the moment we have very vague understanding of both the assumptions and the models. When we do have this understanding, we will likely also have better and simpler models.
Mathematics was always a tool for understanding reality. Proving is understanding. And it was pretty successful so far.
(1) and some other things -- the rest of the planets are far enough, etc. (2) given enough data wrt the size of the tree, under proper distributional assumptions.