There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons in the previous layer and connecting weights" is expressed as matrix-vector multiplication for not really any apparent reason other than it is technically correct and makes for slicker notation, and I guess makes it easier to use APIs that compute gradients for you. Deep learning however is broadly an experimental science, which in many ways is the opposite of math as traditionally envisioned, in which great insights follow deductively from prior great insights. If you ask a basic question like "why should use 4 layers instead of 3?" there is no answer other than "4 works better". Similarly with gradient descent versus random search in weight space. There are many problem domains where random search is as good as any known hill-climbing heuristic search (like gradient descent). Why is GD so effective when learning image classifiers expressed as stacked weight sums? Who knows.