That wasn't my experience. (I went to a top liberal arts college - it didn't give minors, but I took enough math courses that it would've been a minor at other schools.)
In my experience, post-calculus college-level math is qualitatively different from high-school math in focusing on proofs and abstractions over problem-solving, techniques & algorithms. I had an intuitive sense for high-school math. College math kicked my ass, because I was asked to prove everything rigorously, and the proofs just kept coming, dozens of them per homework assignment. That required a very different way of thinking than high-school math did: you needed to remember this large body of theorems and combine them in creative ways to derive new proofs, you couldn't just memorize recipes and techniques that will get you the answer.
I found that physics was more like the continuation of high-school math, except that then your courses had all this lab work and experiment design and fundamental physical principles. One of my friends (a math/physics double major) described it like this: "In physics vectors are objects that live in a vector space, while in math a vector space is a set of axioms from which the concept of a vector and rules for manipulating them naturally fall out."
Edit: hmm, just looked at the username. I know you, and I also know you're better at math than I am. I'm curious just what kind of math you did in high school and what you did in college where the college math was no harder than high school. Either your high school had some really great math courses or your college had some pretty shitty ones. Maybe it's just where we went to school.