"Many people with a large amount of money, perhaps a million or two, consider themselves middle class when clearly they could even retire and live off their passive income"

If you have a million dollars, stop working, invest the money, you can safely draw about 35K per year for a long time. This is based on long term historic returns of the stock market.

There are different definitions of middle class in the US. The first one I found, Pew 2019, said the range was 46k - 137K.

So living off the passive income of a million dollars puts you well below middle class.