The vast majority of "useless" mathematics really do turn out to be useless. In the rare exceptions, there's not much evidence that doing the work beforehand is actually an advantage. E.g. Einstein wasn't aware of most of the work on non-Euclidian geometry before developing relativity IIRC.
Stuff like prime numbers have eaten up millions of brain hours of highly intelligent people. I remember thinking it was weird that so many project Euler problems were about prime numbers. And I looked up what the applications of them were and couldn't find anything significant beyond cryptography.
And they seem to have been chosen for cryptography simply because it was a well studied problem with certain properties. Not because cryptography inherently needs prime numbers and would be impossible without centuries of previous work studying them.