To be pedantic, shouldn't you also need to rule out the possibility that x is a negative number? For example, x = -1 satisfies your quadratic equation, but because we are doing an odd power of x you can rule it out.

You can’t just assume a range of natural numbers in this context. You have a prime p that is equal to f(x) for some polynomial f with integer coefficients and some integer x. You observe that f(x)=a(x)b(x) where a and b are also polynomials with integer coefficients. It does not follow that a and b are non-negative. As an example, let f(x)=x^3-2x^2-3x+6. Then a(x)=x-2 and b(x)=x^2-3. Now you could try to argue that, if f(x) is prime, then one of a(x) and b(x) is 1, so x is 2 or 3, but f(2)=0 and f(3)=6, so f(x) can’t be prime for integer x. But you would be wrong, and, indeed, f(1)=2, which is prime.

You really do have to consider negative factors.