Yes, I believe you do need to have a good math understanding to understand why a model is behaving the way it is and develop intuition into where the problem might be but that comes with experience. If you're looking into ML research then absolutely you need to learn the math but if you intend to support ML as a programmer then there's plenty of space for you to do so