My theory is that people who like math have a reward system that responds well to gaining an understanding on empirical concepts. I have that, and it does drive me to keep studying math. Not that I find it easy though, I don't think I'm able to skip steps, and I often have to repeat things I've already done before they sink in. The difference is that I find this process enjoyable, so I don't mind spending the time.

If I can compare to another activity, I've always wanted to be an artist as well, and have spend quite a bit of time trying to build up the skills. The problem is that, if I'm honest with myself, is I just don't enjoy the process of creative expression, it doesn't trigger any reward system that means anything for me. I wish it did but there's just nothing there. It was a hard pill to swallow, but I realized I like the idea of being an artist, but I don't enjoy the process. Hence my ultimately crummy artwork!

Sorry, I realized I'm talking about myself more than you, but I hope it's some help. The point I hope it makes is that everyone has a different personality, and from that different reward systems. It sounds to me like yours doesn't align with math, and that's fine. I wouldn't try to force yourself to study something which you don't love, at least if it's optional self study. Find subjects that you love learning, and the results will come naturally.