Progress in math often (or usually) isn’t driven by physical problems, so there’s that. Just like progress in physics often (or usually) isn’t driven by “real world” problems. Discoveries are often made before applications are found.
As a concrete example, number theory is an incredibly profound and fruitful branch of mathematics. It is also largely “useless” to this day (outside of cryptography, which uses only a small subset of number theory, and isn’t really “physical” to begin with).
Another well-known example is non-Euclidean geometry, which came into being due to the purely intellectual pursuit of the Fifth Postulate, and almost magically found its place in the theory of gravitation long afterwards.
It takes a certain kind of personality to enjoy mathematics as is, even if it seems “useless”.
(Theoretical physicist who double majored in math and physics in college here. Definitely possess the “certain kind of personality”.)