Another field that might be interesting to compare is analog design. There is a similar stack of theories: lumped element -> transmission line -> maxwell's equations. And yet analog IC design depends heavily on inherited mental models from mentors and modifications of well known topologies. Outsiders think it is black magic. The physics is all understood (nearly) perfectly, and yet knowing the details of QM that explains MOSFET operation helps not at all (or very little) when designing actual useful circuits. The real world considerations of parasitics, coupling, etc. dominate, and extensive formal analysis is not terribly useful. The general methodology is to make changes to the design based on intuition, simple predictive models that give you a direction, and previous experience, and then simulate to see how you did.

A ton of high-quality engineering is done based on intuition, mental models, and patterns learned over years of experience. My hunch is that deep learning will be the same.

EDIT: Just reread, and I want to clarify. I'm not saying that analog design is at the same stage of development as deep learning, or that it is anywhere near as ad hoc. Deep learning probably has a long way to go, but it could potentially end up in a similar state where years of experience is critical and intuition rules.