Quite possibly, yes.
Let's plug in some plausible numbers. Say someone buys a 3-bedroom house for $300k (they have kids). At the same time, 1-bedroom houses sell for $200k.
Time passes, the kids grow up and move out. Say house prices didn't change at all. You sell for $300k, buy for $200k, pay some agent fees, have $85k or so left over.
Now say prices went up and the 3-bedroom house is $1,500,000. You sell, have a $1.2e6 capital gain. $500k is deductible, you pay taxes on the remaining $700k. That's taxed at 20% federal plus the ACA 3.8% investment income tax (because your income is way over that boundary) plus state capital gains taxes. In California, capital gains are taxed the same as other income, at first glance; let's be charitable and assume you avoid the higher tax brackets (12.3%) and get taxed at an effective 10% here. That's 33.8% tax. So your after-tax sale proceeds are $500k + 0.662*$700k = $963k.
Note that your house went up in price by 5x. If the smaller one did too, it costs $1 million and you can't pay for it anymore. If it went up by more than that (e.g. if the primary inflation was in land prices, which it often is) you might be even more in the hole.
Things are even worse if you're trying to sell and buy something equivalent instead of downsizing.