Now, 8+ years later, I wish I had stuck with it or found better instructors or a better school or something.
To throw that opportunity away in exchange for a much less interesting or useful degree is something I still regret but am trying to move on from... can't change the past.
The times I was successful in tough math classes was when I asked for help, so my experience mirrors what others are saying here. The problem was having instructors that sometimes couldn't help much when asked.
I remember having a complete block over lambda calculus - I pretty much locked myself in my room over an Easter break and worked through loads of examples - eventually it clicked and I ended up being really interested in all things functional.
This is larger than math. Successful people ask for help.
[NB The fact I ended up doing research with a bunch of control engineers rather demonstrates that I did manage to get over my fear of maths!].