I wanted to be a physicist really bad. However, I did not have any advanced placement credit in math. I started college taking Calculus I along with the basic intro physics courses. The first year was fine. The second year was largely devoted to quantum mechanics, which is where the trouble started. Semester 1 of my sophomore year, I was taking Calculus III and Differential Equations. Unfortunately, a few weeks into the curriculum, we started using partial differential equations everywhere -- I barely even knew how to solve basic differential equations! Even more unfortunate was the fact that every other student in the dwindling Physics class had advanced placement credit and already had the prerequisite math.
It totally killed the joy I initially found in Physics. I found that I just couldn't make up the lost ground myself, and ended up dropping out of Physics.