Not necessarily. Impressive stone buildings in the Western Balkans were often built by people who were quite poor, except for what livestock they held. It helped that society was clan-based, so resources were pooled, and when these families had to build a house in a region of vendetta and invasion, for protection they chose stone that has held up well over time.

An interesting parallel might be those old Amish houses in the US built well through communal effort, even though the families might have been cash-poor.