Is there a limit to how many times something can be logically divided? If not, then there’s your infinity. It doesn’t require you to continue brute forcing it, just reason about it.
We could redefine "half" to mean "half of whatever you're talking about until you get to some arbitrary limit", but doing that to all of arithmetic is going to wind up in a very odd place.
If you accept that every natural number has a successor which is a natural number, and no two natural numbers have the same successor, and that there’s no loops (e.g. by saying that there’s a total order on natural numbers and that any natural number is less than its successor), then there can’t be a finite collection which is all the natural numbers.
You could say “there’s no collection which has all the natural numbers”, which, ok, how do you want to talk about things true of all natural numbers then?
Formulating descriptions of physics without the axiom of infinity (or, without something to play the role of the real numbers) is super icky. You, in practice, can’t do any significant mathematical physics in an ultrafinitistic approach.
There's an entire branch of math for that: https://en.wikipedia.org/wiki/Constructivism_(philosophy_of_...