To the degree that there's a stigma associated with adults living with their parents in the US, the stigma comes more from being the sort of person who would want to live with their parents when they could afford to be on their own. That's the stigma. It's not a stigma about poverty, it's a stigma about someone who would want a more sheltered and less independent life, and a stigma about someone who wouldn't get seriously annoyed at having their family around 24/7. Those are both pretty alien concepts to a lot of people in the US. There's also the rule of thumb that the #1 way to keep a family together and loving each other and constructively involved in each other's lives is to not see each other every day. Absence makes the heart grow fonder, and all that.
But like you said, this is independent of it being good or bad or right or wrong, it's really just cultural context, but that's the cultural context.