The universe might be infinite, but growth will eventually be limited by the speed of light.

Per capita energy usage is about 6 MWh / year and our population has been doubling every 53 years. Assuming that continues, in 10000 years we'll be using 4e69 Wh / year. If we figure out 100% conversion of mass to energy, that's 1e56 kg of fuel, which is a lot more than all the matter in the visible universe.

You can play with these numbers a little, but even reducing the energy usage per person by a million only gives us another 1000 years. Soon we'll be growing so fast that every year we'll use all the resources within one light-year.