Infinity is a social construct in mathematics. Mathematics values terseness over case analysis, because humans have to manipulate the formulas. By introducing infinities, formulas have fewer cases to analyze.

You can have math without infinity and without undecidability, but there's much more case analysis and things get ugly. Look at how SAT solvers with rational arithmetic work.