Energy source constraints are not the only ones that exist. We also extract many other things from the planet and transform them into other things that we use. These extractions/transformations are also very problematic, and subject to strong constraints. Examples:
* The clearing of forests for farmland for the ever increasing population and increased (unneeded) calorie consumption per person destroys ecosystems, and kills species.
* The growing of massive amounts of food extracts micronutrients from the soil, and these not replaced. This destroys the health of future generations.
* Cement and concrete are made by naturally occurring rocks and minerals, and these are also fast depleting. Mining for more difficult sources also result in pollution and destruction of ecosystems. Pollution also negatively impacts the health of current and future generations, and concrete cities destroy ecosystems.
* Plastics are made from fossil fuels. They break down into microplastics, and leach toxic chemicals into our food and water supplies.
All of these examples involve negative externalities shared by everyone in the world. All of these examples stem from fast and continued growth.