We don't have to give up on infinite sets, but instead think of them as limiting processes instead of a complete object. Basically sacrifice the notion of "set" as a realized object. Just like sqrt(2) is not a rational number, but it is a computable number, which means that there is a computation that can yield an approximation that matches reality to whatever precision is requested.
Similarly the natural numbers are a process that can produce more natural numbers, and can do so for however large a limit you choose.
The problem of rearranging the elements of a set to get a different sum is handled here because you are looking at the limit of the sum, not the sum of the abstract set. Just as in classical math you have to make sure that to compute { sum_i=0^inf (-1)^i / i } that you are actually computing { lim N->inf sum_i=0^N (-1)^i / i }, with intuitionist math you can't help but do the latter, because the former statement makes no sense except as the latter.
The "adding an item to a set makes it larger" is only problematic if you're trying to define "larger" for non-terminating sets, and in addition trying to treat them as "sets" divorced from their underlying structure. The rationals and the integers may look the same to a set theorist, but they are clearly very different objects and it's not at all clear what additional power you get from introducing morphisms that don't preserve any underlying structure.