Sorry, just to be clear, they are obscure, not secret. Anyone can compute primes. No prime is secret.
Otherwise, reductio ad absurdum, Following the same line of logic, all cryptography is "security through obscurity" because there exists a secret key, which is not incorrect. The practice of hiding the design of a system is obscurity, the practice of hiding a specific key to a ciphertext is security, not obscurity, as long as the method of generating and using it is publicly available. If there's nothing to reverse engineer, there can be no obscurity. The method of finding a symmetric key or finding a RSA key is well-known, not obscured, it's simply the size of the problem is not computationally tractable in practice by today's computers.