Mainly because that means there is finite growth in the economy, and we've built a ponzi scheme of sorts at this point. It's a hard pill to swallow, that we can't just grow forever.
Also, economic growth does not necessarily rely on increased production of physical goods. If people stopped paying for physical toys in lieu of software, for instance, the economy very well could grow while the physical footprint of the average person decreased (all else remaining the same). If people preferred experiential entertainment such as movies, escape rooms, etc more than e.g. boating, that could also result in economic growth while physical consumption of goods decreases.
But, while we can continue to grow economically without increasing physical consumption (all else being equal), the elephant in the room is that a very large portion of the world does not live a life even remotely similar to that of a wealthy, developed country resident wrt physical resource consumption. So even if we in rich countries maintained our resource usage, global demand for physical resources will continue to grow rapidly as other parts of the world develop. So are we willing to consign huge swathes of the globe to eternal poverty, or will we actively decrease our resource consumption?
A good short term thing would be to more clearly understand the real costs of the things we do today. What is the long term environmental impact and so on.