edit: after noticing this other hacker news article (https://news.ycombinator.com/item?id=21107706), I wanted to add that this line of thinking is applicable to understanding programs and proofs written by humans as well. Programs and proofs can be well-understood when their pieces, and the way those pieces compose, are well-understood. When the pieces, e.g. lemmata in a proof, are large or hard to decompose, the proof (i.e. the solution to a problem) is harder to verify and understand.
That said, in general I don’t expect that we could understand any particular solution produced by an AI, be it deep or otherwise, but I do expect it to be possible quite often.
The holy grail of neural nets has always been to build a simulation of the brain, figure out how it works, and apply that knowledge to how the human brain might work.
We're not there yet but progress has been made. Eventually we'll understand NNs well enough to explain not only themselves but also human brains. In any case we have no choice because we cannot deploy NNs in life critical situations until we understand how they work, because that's the only way to understand how they fail.
I'd say that's a goal for some people -- for those whose goal is to figure out how the brain works, rather than constructing a more ideal and powerful GI. Remember the brain is great at some things, but laughable at others -- such as a "7 +/- 2" items in short term memory, inability to immediately retain rote knowledge after one instance and in great numbers, etc. It's the merging of the fuzzy, goal-directed behavior of the mind, in conjunction with its ability to effect the "real world", and the super-human memory and computational capabilities of computers that makes possible future GAIs that are so powerful and possibly scary.