Imagine an economy in perfect stasis, no growth, no shrinkage, just the same stuff year after year. Now introduce some efficiency improvement, like shaving some weight from a car design, or a computer chip that performs the same computation for less power, or an agricultural technique which requires less land for the same yield. Voilà, growth without consuming additional physical resources.
Of course, this is not our main source of growth at the moment. But surely it could be?