We don't know that the physical world has infinite precision. In particular, time doesn't even seem to be continuous as far as we can tell (cf. Planck constant). And we think there's a finite amount of mass in the universe, so how could we encode arbitrarily large natural numbers (as is required to model Peano arithmetic)?