According to Susan's account, it took her 1.5 years to go from no college-level math to a full college-level physics curriculum. At the school I studied at, the Calc series (I, II, and III) takes 1.5 years. The intro physics (I and II) takes a full year, and the upper physics take at least a year (Mechanics I & II, E&M I&II, Relativity I&II, Quantum I&II, Lab I&II). (And you had to take intro physics before higher-level physics, and most courses had dependencies.)
That's approximately 45 credit hours. If you took 18 credits a semester, that would take at least 2.5 years. And those are all tough courses -- no 3 credit gen ed "filler" courses. Even at 15 credits a semester, that's still 3 years.
So, to get through all of those courses in 1.5 years would be like learning about 30 credit hours / semester. And add in the physics electives, like thermo, waves, statistical mechanics, quantum field theory, additional math courses, etc... that's a lot on top.
I guess my point is that if Susan's account is discouraging, note that a lot of people take far more than 1.5 years to get through undergraduate physics.