Yes, personal experience and watching other people around me as well. Mostly, hard is boring, i.e. when you don't have enough interest to pursue it. As for math - it is not just about being good, but also the process itself - of becoming good, i.e. learning. When this process seems interesting and fun, I doubt someone will say it is hard.

E.g. for many people, literature classes or some other humanities stuff is hard.