He's saying that, in the places where people do work harder, they don't actually have more/larger/better houses. Rather, the work just becomes part of a zero-sum competition that bids up the price of the (fixed) quantity/quality of houses.

This partially contradicts your point.

What I would add (to reconcile the two points), is that one kind of work is not fungible with another kind of work. Yes, people work very hard in Silicon Valley -- but they are not working hard at building houses. If they were, there'd be a lot of supply, and the price would fall.

Overall, this is perhaps a comment about the (mis-)allocation of work in society.