Yeah but for sufficiently large numbers with sufficiently large factors this is not a computationally easy task.

(RSA encryption, for example, relies on factorisation being hard; I can only presume OP was inspired by recent news regarding possible developments there.)