That's why I loved it. But I think this is a dilemma of teaching any number of subjects: How to convey both the useful aspect and the intrinsically interesting aspect, for lack of a better term.
The thing is, once something is applied, it has to deal with all the complexity and uglyness of the real world. But if you stick to abstract ideas, you are only limited by your imagination and logic. The mundane aspects of an applied field can detract from the pure beauty of an abstract subject.
I am not saying that this is always true, and sometimes it is nice to see applications of some abstract concepts. But many people intentionally shy away from applications towards a more theoretical field.
But you kind of get to a point after a couple years or so where you've learned enough math to handle the physics coursework -- including being able to pick up new math topics quickly as needed. And that's also the point, like 3rd year, where math really begins to come alive as an end unto itself. I loved the abstract stuff and the beauty of proofs, and it was as much of an escape from reality as a way to engage with it.
Edit: I imagine you were just tongue in cheek, though.
While you could construe the negation to be what you propose, it's not uncommon to read it as "not having to have a use in mind".
The part I hated about maths was all the memorization required ;)