Of course it is a concern. No nation survives with an indefinite dwindling population. If nations turn to immigration as a way to bolster their population, it cones with its own set of problems.
China, with a population of 500 million, would be a much more pleasant place to live. China with a population of 500 million Han and 800 million-strong hodge-podge of other groups constantly resenting each other and together resenting the Han? Not so nice.
Things are never monotonically increasing or decreasing forever, and there is no reason to prevent population from adjusting up or down in order to reach better conditions for a given environment. Nor do we need to set public policy to focus solely on maximizing total GDP and real estate values.