Our current population is being propped up by all number of finite resources, most significantly oil, for which we have no certainty of finding more supply of in the future. With a higher population, we're depleting those resources faster.
It may in fact be better to get to a lower level of population which is long-term stable. We are aggressively burning through our natural resources and while humanity won't go extinct our population is doomed to absolutely plummet during an age of poverty.