Definitely, in fact Cryptography was added on as a trailer to my Discrete Math course.
Yeah it's a building block for understanding number theory well, which is important in cryptography.
One cool thing about math though is that continuous/analysis concepts and discrete concepts interact with each other. There's a fair bit of calculus in e.g. AoCP. At some point number theory as well. Kind of depends on how deep you want to go. You can learn a lot about cryptography without really needing a very strong mathematical base.