The ease of doing it in class or breezing through homework assignments was backed by hours a night of looking at high level concepts and talking about math with people more advanced than me, including teachers during lunch breaks and after school - a habit I'd had since late elementary school.
At the end of the day, they weren't "bad at math", I just put a lot more effort in to my practice - and it showed on game day, as it were.
Just like the guy in my school who was "gifted" at basketball spent hours a night practicing since early elementary school.