Atoms are tightly bound and hard to split, they are even harder to build. Up one level and starting with atoms the ease with which the next structures (molecules) can be created goes up tremendously (of course with a lot of variation depending on which atoms are used).

Prime numbers are can only be split (factorized) in one axiomatic way (1 and the prime itself). This is with multiplication(or it's inverse division) as the building (splitting) operation. Primes are tough to deal with, yet with them you can create (through multiplication) all other numbers. This is the fundamental theory of arithmetic.

The correlation is apparent. Though as explained with atoms we are creating structures that are made up of atoms but are not classified as atoms themselves (fusion can give you atoms from atoms but it is not easy business), but with primes numbers we make other numbers that are not primes.