>But we can skip out ideally half of the numbers.

ideally, but in the worst case it's O(n * (n - 1)), or O (n^2)

if your first X is the max, and your second choice is X-1, and so forth (in the worst case), then you're not operating in O(n), but n^2