Sure, but we have concrete buildings still standing from Roman times - most famously, the Pantheon.

I grew up in a cottage a couple of hundred years old, with thick walls built from stone and mortar. It's not unusual to live in a house that's over a century old in Europe.

I've also lived in a wooden house - more of a three room + toilet chalet, really. Biggest irritations were all the noises from thermal expansion in the summer, and the fact rats could eat their way in. I think it's a bit more effort to actively maintain wood and be sure it's in good condition than brick and concrete construction.

That cottage, btw - the roof had a wood frame, but it wasn't the original - that had burned down along with the thatch at least once, and been replaced some time before we moved in.