Middle and working class grown ups with families don't care about hot tubs and parties, but the things they struggle to afford: decent schools for their children, their mortgage, etc, are things that he never would have to worry about. His decision to live in Manhattan with less stuff doesn't change that reality for them, and I question whether it really puts him in touch with their reality.
Bringing up his personal life choice detracts from the valid point that the wealthy can isolate themselves from the concerns of everyone else if they choose to.
And his argument about wealth vs. smarts is attacking windmills ... nobody seriously believes that having more wealth makes you smarter.
Edit: lost second half, re-added. Edit2: wording tweak