One of the problems is that many of our society's systems are predicated on a growing population. Social security and pensions, for example, are structured not unlike a pyramid scheme: for every old person we should have more than one working young person. People take more than they give. Fixing that will be painful, but possible.
More worrying is how many countries' birth rates have fallen below the replacement rate. Some SE Asian countries are interesting case studies here (Japan, S Korea), but it's not looking good, and much of western Europe is heading in the same direction. Maybe the worry is overblown and populations will eventually stabilize at a lower point, but currently it seems like a declining population will just add to the stressors that are putting people off from having children, so it could just as well keep snowballing.
All that's to say, I don't worry too much about over/underpopulation, but I do worry about a shrinking population.