Where I'm from -- Northern California, USA -- "built like a brick shithouse" was used frequently, and exclusively, to describe very sexy females. It had nothing to do with muscularity per se, and would never have been used to describe a man, not even in jest.

At least, in the 80's it was thus.