I think you're working with a very different definition of "explainable" than is commonly used in the machine learning literature.
It really shouldn't surprise you that we have to go back to the early parts of the field; his argument restated is "machines can't make discoveries, only humans can" and this confused thinking is something that gets addressed in the early writings much moreso than the latter writings.
Frankly, we've known algorithms as far back as Aristotle that were capable of discovery. Peter Norvig in AIMA cites Aristotle when talking about some search problems and path discovery is still discovery. Some of these algorithms literally include frontier in the algorithmic description; clearly a framing that implies thinking of them in terms of discovery. Honestly, what is really happening in my estimation is some people do this thing where they pretend AI is categorically reduced to the subset of things which they can criticize. It is a really weird tendency, but extremely common. You can see it in the persons post actually. Look at the contortions they go through to only focus on supervised learning - to only focus on a narrow subset of supervised learning restricted to deep learning - to only focus on a narrow problem domain of supervised learning - classification - and to only focus on a specific example of that problem domain (we can give good explanations for smaller models) - the ones that humans can't easily explain.