I just felt like we never got to the heart of the matter of why the models work and how to approach developing them, it was all about learning a bag of tricks.
Meanwhile, math and CS being a lot more axiomatic by nature, they also made a lot more sense to me.
That being said, that specificity of physics, the unbridgeable gap between reality and the models we build to describe it, in retrospect, is what makes it more interesting to me today (it's not just a "closed" system in the sense that math is — of course the relationship between math and physics is itself fascinating but that's yet another topic), but I still feel like I haven't found the right pedagogical approach to make it fit my mindset.