The downside is that our entire world economy is predicated on the assumption that labor will be cheap and markets ever expanding. Even if the population stabilizes, it looks like it will be well below what is considered replacement level and so this will require a massive shift in our thinking about how we organize literally everything in our society.
I don't know if it is possible to overstate how much things will change. The service staff you interact with at Starbucks, the grocery store, who deliver your meals, will no longer be affordable. These places will still exist in some form (though there will likely be fewer) and they will be much more automated than they even are now. These people will be so needed elsewhere that salaries will likely draw them to other industries critical to society continuing to function. Healthcare, in particular, will swallow up as much of the population as it is able to (much more than even now in a relative, not absolute sense).
We will have to make hard choices about continuing to support things like suburbs and other far-flung infrastructure that only barely made sense as long as you assumed more and more houses would be built and that property taxes would continue to go up to support it all. That will no longer be true, those houses will start to go empty, then the surrounding commercial properties. There will not be anyone interested in buying them. Some will be torn down, many will probably rot in place. This has already been in progress for decades in economically depressed areas, particularly the so-called "Rust Belt" in the United States.
There will (and already are) massive brain drains in key industries where skilled older workers are either not being replaced, or by those with much less directly applicable experience. The ability to maintain our infrastructure will become strained to the point where we will have to ask ourselves how we can safely reverse our maintenance and operation commitments alongside a constantly shrinking population while avoiding single points of failure that could trigger a panic or societal collapse. This is not far-fetched alarmism, the way we currently live, we are only a week or two of regional or national power outages away from complete anarchy. So many of our systems are interrelated, that a failure of one or two critical ones can have a cascading effect. The smaller local outages in Texas in 2021 was minutes away from leading to nearly months-long blackouts for this reason.
We will need more redundancy and slack in the system to manage these when they occur. Market capitalism abhors redundancy as inefficient so this will not be highly incentivized to be taken on by any one corporation, leading to governments likely deciding to impose extreme regulation or outright nationalize industries.
Interestingly, the United States may be one of the few societies where things more or less run along as normal, but this is only assuming a continued or increasing permissive immigration policy. The more we go the way of the Japanese, all of the above applies.