If you've got a lock on calculus, programming, and linear algebra, then you've got the skills to understand deep learning. Most time spent working on DL is not in the megamath part, it is in finding good network structures to optimize results.
We are trying to address this soon!