There simply has to be limits to this though, because if there were not, the materials would eventually become (relatively) free.

Think of it this way, if the value of output grows too many magnitudes over the inputs, then essentially the input becomes free, which violates intuition, because those resources are scarce. If the resources are free to me, I will buy them all and raise the price.

Arguing otherwise to me is an indication that you think either the material in question is not actually scarce, or else you realize there is actually a limit to how much value the economy can derive from those resources, but fail to reach the logical conclusion that there is a limit to economic growth.