Have you ever seen a negative number of things?
Infinities are quite different, given that it is by definition impossible to measure whether something in reality is infinite or not unless you have infinite time.
It's impossible to count a negative number of things. It's impossible to count an imaginary number of things, it's impossible to count a matrix of things. It's impossible to count an irrational number of things. These are all fictions that people use because they're useful. Infinity is exactly the same as any of those other concepts.
Also I clearly just demonstrated that you can count negative numbers easily so I don't understand why you find it impossible.