How long have bamboo clothes been a thing? That’s the rayon process isn’t it? Has that been around more than twenty years? Patagonia will sell you work clothes that are almost half wood fiber cloth now, with a mix of cotton and hemp to finish it out.
You could definitely eat chestnut porridge from a chestnut bowl with a chestnut spoon, so the clothing is the main differentiator and if chestnut were here today, I’m sure they’d be using it for clothing.