Interesting, that's not my observation at all. Instead freshman classes were the most grueling experiences, intentionally weed-out classes to try to detect and, well, weed-out early those who wouldn't have the combination of grit, intelligence and organization skills to make it to the end of the degree.
I speak somewhat hyperbolically, it was never _unfair_, but the classes absolutely were paced and had homework/exam loads that you would simply be unable to cope with if you were trying to half-ass it. At my college, Calc I and Physics 1 were the big freshman showstoppers. I tested out of calc 1, but my IB HL was chemistry and as such I had a chemist's disdain (and begrudging respect) for the more academic science of physics, and while the actual topics weren't difficult, there was a LOT of homework and the exams left no stone unturned.
Comparatively, jr+ classes tended to be much more relaxed, focused more on learning the material than ensuring you are willing to do 8-10 hours of physics homework per week.
Every college is different though, I suppose.