The biggest problem is that you lose unique prime factorization. With prime factorization, I get a unique representation of every positive integer. Let's consider a way to write positive integers in "base prime", similar to base 10 or base 2. I'll start counting from 1 and write numbers as a tuple of prime factors. Similar to base 10, "base prime" has an infinite set of 0s that we're leaving out for brevity (e.g. 19 = 0000019), although it's on the right side instead of the left.
1 = () = (0, 0, 0, 0, 0, ...)
2 = (1) = (1, 0, 0, 0, 0, ...)
3 = (0, 1)
4 = (2)
5 = (0, 0, 1)
6 = (1, 1)
7 = (0, 0, 0, 1)
8 = (3)
9 = (0, 2)
10 = (1, 0, 1)
The
i th position in every tuple is the power of the
i th prime in the factorization of that number. So 10 = (1, 0, 1) = 2^1 * 3^0 * 5^1. 84 would be (2, 1, 0, 1) = 2^2 * 3^1 * 5^0 * 7^1. If we have unique factorization, there is exactly one way to write every positive integer like this, and there are many insights we can gain from this factorization. If 1 is prime, then we can write 6 = 1^257 * 2^1 * 3^1, or any other power of 1 we like. We just gain nothing from it.
There are often many equivalent ways to define any mathematical object, and I'm sure there are plenty of ways to define a prime number other than "its only factors are itself and 1". These other definitions are likely to obviously exclude 1. One obvious one is the set of basis coordinates in this "unique factorization" space that I just laid out here. And we're never really excluding or making a special case for 1, because 1's factorization is simply the absence of any powers -- empty set, all 0s, whatever you want to call it.
Keep in mind that "unique factorization" turns out to be very interesting in all sorts of other mathematical objects: rings, polynomials, symmetries, vector spaces, etc. They often have their own notion of "prime" or "primitive" objects and the correspondence with integer-primes is much cleaner if we don't consider 1 prime.