I think people like to say the prime numbers are the basic building blocks of arithmetic (or numbers), and we say this because of the Fundamental Theorem of Arithmetic (which says that every natural number has a unique factorization of primes).