As someone with graduate level math experience and many friends in mathematics grad school, I know what you mean, but it definitely depends on your idea of fun.

You have to enjoy bashing your head against a problem for a dozen hours. Or to be more precise, you have to enjoy the feeling of solving such a hard problem sufficiently much to find math fun.