Then I would counter with:
We traveled 1/2 of the way, so we still have 1/2 to travel.
Since we travel at constant speed the 2nd half of the entire way cannot take longer to travel than the 1st half of the entire way.
So each subdivision can NOT ADD more time than the previous 1/2 interval.
So we have an upper bound on time, since time can never be larger.
So while we do have an infinite amount of time intervals, the sum will never grow, it's upwards limited.
But since each "parent interval" is upwards limited, it's irrelevant to how many "child" intervals you subdivide it as the parent time interval is always upwards limited.
So no matter how many times you subdivide, the TOTAL TIME never grows, thus time is not infinite.