1. Both the growth in population and the current rate of decline are 'unprecedented' in velocity. They are 'spikes' like the spike we see in climate change graphs.
2. 'Growth' to the extent there are resources to grow is a positive state, 'shrinkage' even given positive conditions in which growth is possible is a much worse state, because it's never happened and implies something might be wrong - like the possibility of a social or some other kind of disease.
3. 'More' population, given resources to sustain it is at least on 1st order basis 'better' - for obvious reasons. Clearly 'more' (or even present amount) may not be ideal but you'd have to do a second order assessment there (i.e. possibility of long term exhaustion of resources etc.) - and determine ways in which it would be worse, but I don't see any really any reason to believe that 7B is 'less safe' than 500M.
In the event of a major nuclear war, it would probably be better to have more people, of more variety, in more places, in more variety of living conditions.
4. 'Safety Factors' If we could magically calculate some kind of ideal population size, we'd have to assess for all sorts of black swan events (war, famine, disease) and have ample extra population to account for those possibilities. You carry more fuel than you need to cross the Atlantic if you really want to make sure that you get all the way.
4. "What does it matter if there are only 100 million people?"
Ok - so this should have been question #1 because this is they key understanding.
Civilization is a pyramid that generates disproportionate wealth from those operating at higher levels of sophistication. Those people require the support of others doing basic work.
If there were only 1000 people on planet earth, civilization would collapse entirely, everyone would be working on the farm. Only at a certain scale would basic foodstuffs be available such that others could start working on 2cnd order things, like carpentry and basic metallurgy. Significantly more people would be required to have enough slack labour to provide for basic literacy and education. The economy would have to be even larger to start to support the most basic technological advancements.
Etc. Etc. Etc..
Given that we now have a lot of knowledge and have harnessed fossil fuels which provide at least tremendous energy, it would be possible to maintain our state of civility with less people surely, but still quite a lot, definitely more than 100M.
Our economy, which sustains our lives to very high material standards, depends upon an huge amount of sophistication and knowledge. I don't think the number for maintaining that is less than 1B and realistically, given that social systems develop at different paces, and some collapse etc. it's probably more around 2B.