Depends what you mean by hardest. Is it worse to lose 10 million dollars but still have 15 million dollars remaining to console yourself with, or is it worse to lose your job that pays $50k?

The wealthiest have minimal need to sell and asset when it's down. So the magnitude of the hit is the biggest, but if you don't use some of your wealth for diversification and the decent financial advice to get you there, you're just squandering wealth.

In absolute terms, the super rich would be hit hardest. In relative terms, the super rich can afford to take a lot of probably losing plays that protect their assets.

Imagine if the super rich spend 2% of their assets to hedge risk (options, etc.). The average middle class person has most of their assets tied up in their house, and probably cannot afford to lose 2% of their asset base to protect against downturns. Certainly, there are few easily available hedges for a decline in the residential real estate market in (random town/city) as opposed to the easy ones for wealth held in stocks.