I don't think the current models suggest the universe is infinite. As far as I understand, there is a finite amount of atoms supposed.
Also, there are very clear examples of physical limits to growth. For example, gravity and its interaction with the other forces prevents animals, buildings, trees, stars, etc. from growing too big. What happens if they grow too big? Collapse.