Ask HN: What are the open questions in understanding Neural Nets?

Dear HN community,

I often read (on the web or in papers) that Neural Nets are great for a variety of machine learning tasks, yet their workings are not fully understood.

I'd like to know what exactly are the open problems in understanding NNs. Are these questions mostly related to training? Inference?Anything else? Can it be boiled down to one central question?

I hope someone smarter than me can shed some light on this.