Can you elaborate on this, and at what level? Are you talking about a PhD level of understanding of cutting edge mathematics, or do you mean understand the basics, or somewhere in between?
1. Asking a candidate to cast a non-standard problem of classification or estimation into a tractable optimization problem. (this is a very valuable skill that someone who has done good studies in numerical linear algebra/machine learning/stat/information theory/control systems/signal processing/math/etc. should be able to do)
2. Asking them to take an algorithm they have used and explain every step in deriving the algorithm. (It will help interviewer calibrate the level of learning in the interviewee. Also helps screen for indisciplined black-box users.)
3. Presenting challenging machine learning scenarios: using customized ensemble learning approaches, imbalanced data sets, noisy labels, multiple instances, different error metrics, etc. and seeing how interviewee approaches the problem from first principles (real world problems almost always involve some of these issues)
4. Testing their intuitions in "feature-engineering" for different types of data. (with the partial exception of cases where rigorous research/successful products show the utility of deep learning, one has to almost necessarily do a fair bit of feature-engineering)