Potential infinity aligns with the explanation you provide in the second paragraph. It refers to the process of arbitrarily unbounded enumeration (having the potential to count to any arbitrary number). This is equivalent to the capabilities of a Turing Machine. Aristotle's argument is that the human mind (Turing Machine) is restricted to computing decidable problems and cannot compute undecidable ones. The whole point of the application of Cantor's diagonalization argument is to demonstrate this limitation.
Cantor rather argues that assuming the Axiom of Infinity does not necessarily lead to a contradiction (unless you assume the opposite of course). The assumption simply states that some infinitely large set exists, in particular, the natural numbers. It does not have to physically exist, but we certainly can theoretically associate a finite characterization to it. You should look into Kolmogorov complexity for this. Note that this is NOT at all the same thing as "counting to infinity".
A lot of people will counterargue that infinity is just a concept, but I feel as if they miss part of the point. A similar argument would lead to the conclusion that pi does not exist and neither does the number, 2. The only difference is that we apply the concept of two-ness to discrete objects we can compute with. We do have recursive descriptions (programs) that can describe a countable infinity (aleph null) or even pi. We might as well use these descriptions as placeholders for the actual thing. While we cannot contain the base 10 encoding of pi, we have another encoding of it of finite length (the program). Who is to say that a base 10 encoding of numbers is a better proof of existence than one written in C++?
You might have more success with arguing against the existence of undefinable numbers. These do not have a description of finite length and are definitely numbers that we cannot conceptualize with our current assumed limitations.